-- Logs begin at Sun 2022-10-30 00:10:42 UTC, end at Sun 2022-10-30 01:21:39 UTC. -- Oct 30 00:42:07.644086 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Oct 30 00:42:07.653188 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Oct 30 00:42:07.655139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: *** Starting uWSGI 2.0.18-debian (64bit) on [Sun Oct 30 00:42:07 2022] *** Oct 30 00:42:07.655139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Oct 30 00:42:07.655139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Oct 30 00:42:07.655139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: nodename: dv-cinder-master-3-iscsi Oct 30 00:42:07.655139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: machine: x86_64 Oct 30 00:42:07.655139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: clock source: unix Oct 30 00:42:07.655139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: pcre jit disabled Oct 30 00:42:07.655139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: detected number of CPU cores: 4 Oct 30 00:42:07.655139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: current working directory: / Oct 30 00:42:07.655139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: detected binary path: /usr/bin/uwsgi-core Oct 30 00:42:07.655139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: your processes number limit is 47680 Oct 30 00:42:07.655139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: your memory page size is 4096 bytes Oct 30 00:42:07.655139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: detected max file descriptor number: 2048 Oct 30 00:42:07.655139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: lock engine: pthread robust mutexes Oct 30 00:42:07.655941 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: thunder lock: enabled Oct 30 00:42:07.655941 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Oct 30 00:42:07.655941 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Oct 30 00:42:07.694205 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: Python main interpreter initialized at 0x55d66c5f99c0 Oct 30 00:42:07.694205 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: python threads support enabled Oct 30 00:42:07.694205 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: your server socket listen backlog is limited to 100 connections Oct 30 00:42:07.694205 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: your mercy for graceful operations on workers is 90 seconds Oct 30 00:42:07.694623 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: mapped 403077 bytes (393 KB) for 2 cores Oct 30 00:42:07.694623 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: *** Operational MODE: preforking *** Oct 30 00:42:07.694859 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: *** uWSGI is running in multiple interpreter mode *** Oct 30 00:42:07.694859 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: spawned uWSGI master process (pid: 81486) Oct 30 00:42:07.695067 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Oct 30 00:42:07.696080 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: spawned uWSGI worker 1 (pid: 81494, cores: 1) Oct 30 00:42:07.696080 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: spawned uWSGI worker 2 (pid: 81495, cores: 1) Oct 30 00:42:07.696080 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Oct 30 00:42:11.874180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 30 00:42:11.874180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: from cryptography.utils import int_from_bytes Oct 30 00:42:11.875001 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 30 00:42:11.875001 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: from cryptography.utils import int_from_bytes Oct 30 00:42:11.875111 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 30 00:42:11.875111 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: from cryptography.utils import int_from_bytes Oct 30 00:42:11.875111 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 30 00:42:11.875111 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: from cryptography.utils import int_from_bytes Oct 30 00:42:11.923489 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 30 00:42:11.923679 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 30 00:42:11.923801 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 30 00:42:11.923879 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 30 00:42:11.923943 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] command line args: [] {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 30 00:42:11.923999 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] command line args: [] {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 30 00:42:11.924141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 30 00:42:11.924218 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 30 00:42:11.924282 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ================================================================================ {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 30 00:42:11.924425 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ================================================================================ {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 30 00:42:11.924566 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] admin_token = **** {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.924653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] admin_token = **** {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.924738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] config_dir = [] {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.924904 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] config_dir = [] {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.924996 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.925057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.925129 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] config_source = [] {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.925190 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] config_source = [] {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.925258 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] debug = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.925458 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] debug = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.925591 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.926906 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.927009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.927078 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.927136 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.927192 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.927252 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.927315 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.927382 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.927446 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.927507 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] list_limit = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.927567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] list_limit = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.927629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] log_config_append = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.927690 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] log_config_append = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.927959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.928043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.928102 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] log_dir = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.928162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] log_dir = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.928231 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] log_file = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.928289 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] log_file = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.928353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.928415 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.928471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.928526 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.928580 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.928639 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.928704 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.928768 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.928832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.928893 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.928893 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.929002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.929058 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.929115 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.929176 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.929233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.929298 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.929363 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.929427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.929573 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.929647 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.929754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.929874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.929960 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.930126 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.930306 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.930393 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.930471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.930593 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.930678 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.931071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.16/identity {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.931203 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.16/identity {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.931296 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] publish_errors = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.931369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] publish_errors = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.931502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.931588 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.931659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.931818 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.931897 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.932010 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.932082 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.932210 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.932294 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.932367 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.932529 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] standard_threads = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.932651 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] standard_threads = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.932728 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.932815 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.932977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.933108 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.933188 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.933258 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.933329 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] use_journal = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.933451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] use_journal = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.933530 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] use_json = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.933606 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] use_json = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.933683 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] use_stderr = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.933752 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] use_stderr = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.933823 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] use_syslog = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.933900 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] use_syslog = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.933971 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.934044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 30 00:42:11.934235 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.934356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.934433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.934516 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.934708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.934829 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.934913 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.934985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.935147 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.935286 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.935397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.935649 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.935835 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.935957 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.936073 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.936201 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.936314 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.936430 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.936550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.936666 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.936775 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.936934 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.937048 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.937158 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.937266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.937374 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.937482 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.937920 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.938030 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.938140 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.938248 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.938356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.938464 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.938579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.938689 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.938806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.938915 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.939034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.939229 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.939348 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.939458 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.939577 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.939686 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.939849 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.939969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.940084 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.940200 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.940309 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.940418 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.940528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.940647 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.940756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.940869 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.940984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.941093 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.941201 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.941318 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.941427 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.941534 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.941644 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.941752 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.941859 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.941974 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.942083 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.942198 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.942308 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.942415 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.942522 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.942638 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.942754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.942862 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.942976 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.943084 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.943191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.943309 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.943419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.943611 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.943743 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.943855 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.943975 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.944094 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.944094 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.944432 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.944549 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.944659 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.944767 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.944952 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.945105 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.945235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.945343 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.945459 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.945568 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.945688 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.945797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.945905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.946012 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.946119 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.946225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.946344 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.946453 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.946562 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.946673 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.946782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.946894 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.947048 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.947147 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.947239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.947329 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.947420 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.947519 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.947614 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.947710 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.947849 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.947981 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.948091 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.948213 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.948326 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.948435 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.948546 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.948656 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.948764 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.948872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.948981 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.949089 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.949203 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.949322 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.949441 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.949550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.949660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.949780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.949881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.949995 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.950105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.950213 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.950313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.950428 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.950546 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.950649 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.950757 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.950866 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.950974 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.951083 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.951192 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.951306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.951407 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.951514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.951623 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.951759 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.951873 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.951982 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.952090 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.952199 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.952308 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.952429 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.952539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.952654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.952771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.952881 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.952990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.953107 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.953209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.953317 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.953426 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.953541 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.953651 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.953779 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.953891 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.954004 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.954114 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.954222 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.954331 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.954451 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.954562 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.954671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.954780 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.954889 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.954998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.955115 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.955232 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.955342 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.955460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.955570 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.955679 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.955912 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.956027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.956143 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.956253 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.956363 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.956480 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.956598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.956722 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.956835 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.956951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.957099 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.957212 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.957328 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.957446 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.957563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.957673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.957782 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.957890 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.957999 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.958116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.958234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.958352 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.958462 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.958571 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.958680 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.958797 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.958907 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.959017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.959162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.959284 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.959421 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.959541 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] auth.external = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.959652 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] auth.external = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.959787 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.959900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.960010 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.960123 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.960241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.960353 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.960470 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] auth.password = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.960582 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] auth.password = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.960691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] auth.token = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.960801 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] auth.token = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.960910 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.961020 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.961138 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.961251 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.961360 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.961471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.961580 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.961689 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.961806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.961918 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.962028 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.962138 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.962248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.962358 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.962475 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] credential.caching = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.962586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] credential.caching = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.962696 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.962806 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.962915 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.963024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.963145 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.963258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.963367 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.963477 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.963586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.963695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.963870 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.963983 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.964100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.964212 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.964322 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.964433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.964551 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.964662 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.964772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.964890 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.965002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.965112 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.965230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.965342 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.965452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.965562 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.965672 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.965781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.965905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.966017 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.966127 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] federation.caching = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.966237 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] federation.caching = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.966347 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.966466 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.966577 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.966691 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.966802 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.966911 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.967027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.967181 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.967296 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.967407 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.967519 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.967629 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.967752 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.967877 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.967989 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.968099 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.968208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.968318 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.968428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.968547 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.968658 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.968776 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.968886 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] identity.caching = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.968997 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] identity.caching = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.969106 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.969297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.969410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.969521 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.969632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.969741 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.969850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.969968 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.970079 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.970189 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.970307 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.970418 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.970528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.970646 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.970757 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.970868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.970978 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.971094 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.971205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.971323 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.971434 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.971545 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.971664 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.971810 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.971922 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.972044 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.972157 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.972266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.972376 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.972486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.972596 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.972716 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.972828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.972938 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.973049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.973159 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.973268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.973386 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.973497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.973607 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.973717 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.973828 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.973938 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.973938 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.974155 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.974264 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.974373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.974373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.974578 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.974696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.974838 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.974951 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.975060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.975060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.975354 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.975354 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.975563 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.975563 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.975785 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.975898 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.976021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.976021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.976229 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.976339 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.976449 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.976562 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.976683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.976795 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.976906 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.977015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.977125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.977267 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.977388 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.977501 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.977610 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.977719 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.977828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.977938 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.978056 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.978168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.978278 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.978387 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.978497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.978607 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.978725 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.978837 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.978946 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.979057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.979166 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.979284 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.979404 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.979515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.979625 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.979781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.979896 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.980006 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.980129 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.980241 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.980351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.980460 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.980569 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.user = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.980679 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.user = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.980796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.980908 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.981017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.981128 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.981237 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.981353 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.981472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.981584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.981693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.981810 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.981927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.982038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.982156 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.982267 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.982376 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.982486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.982595 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.982705 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.982822 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.982935 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.983044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.983153 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.983263 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.983373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.983490 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.983601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.983710 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.983841 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.983951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.984060 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.984182 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.984301 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.984411 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.984521 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.984631 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.984740 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.984859 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.984970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.985079 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.985196 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.985308 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.985418 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.985535 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.985646 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.985756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.985865 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.985974 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.986083 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.986201 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.986318 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.986429 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.986538 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.986648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.986761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.986880 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.986991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.987103 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.987213 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.987353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.987467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.987590 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.987702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.987843 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.987956 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.988066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.988176 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.988294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.988405 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.988515 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.988625 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.988734 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.988849 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.988968 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] resource.caching = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.988968 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.989175 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] resource.caching = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.989290 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.989401 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.989520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.989631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.989740 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.989850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.989965 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.990075 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.990194 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.990305 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.990414 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.990524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.990633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.990743 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.990934 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.991046 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.991163 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] role.caching = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.991274 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.991383 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] role.driver = sql {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.991493 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] role.caching = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.991612 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.991737 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] role.driver = sql {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.991837 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.991937 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.992045 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.992155 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.992277 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.992420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.992538 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.992650 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.992759 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.992877 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.992999 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.993114 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.993224 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.993333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.993442 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.993551 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.993669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.993786 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.993897 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.994006 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.994122 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.994232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.994349 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.994462 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.994572 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.994682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.994792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.994901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.995019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.995129 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.995239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.995349 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.995458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.995567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.995685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.995854 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.995966 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.996076 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.996186 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.996308 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.996419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.996529 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.996639 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.996754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.996865 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.996983 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.997096 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.997096 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.997303 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.997443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.997556 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.997556 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.997772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.997884 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.997993 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.998102 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.998212 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.998330 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.998443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.998443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.998649 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.998757 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.998859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.998977 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] token.caching = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.999088 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.999198 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.999308 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.999417 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.999526 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] token.caching = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.999644 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.999767 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.999767 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:11.999975 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.000085 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.000195 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.000195 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.000415 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.000524 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.000634 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.000743 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.000852 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.000973 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.001090 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.001199 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.001310 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.001419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.001528 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.001645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.001756 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.001865 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.001974 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.001974 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.002179 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.002179 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.002403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.002522 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.002633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.002742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.002852 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.002852 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.003076 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] database.connection = **** {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.003076 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.003292 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.003403 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.003512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] database.connection = **** {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.003621 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.003761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.003874 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.003996 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.004111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.004221 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.004329 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.004439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.004547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.004665 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.004776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.004885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.004994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.005103 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.005214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.005331 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.005443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.005552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.005661 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.005770 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.005880 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.005997 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.006143 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.006254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.006364 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.006473 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.006583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.006705 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.006819 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.006931 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.007041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.007041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81495) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 30 00:42:12.007247 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.007247 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.007498 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 30 00:42:12.007611 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81494) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 30 00:42:12.036450 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.036598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.036732 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.036813 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.036875 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.036994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.045265 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.045446 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.075177 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.075338 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.075408 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.075527 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.126379 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.126668 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.173889 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.174118 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.175013 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.175249 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.329348 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.329555 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.329685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.329770 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.329846 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.329936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.330014 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.330076 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.330150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.330319 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.330465 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.330555 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.353436 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.353618 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.353705 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.353792 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.353870 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.354037 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.354166 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.354303 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.354439 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.354522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.354585 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.354666 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.354740 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.354824 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.365917 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.366138 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.378649 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.378783 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.379005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.379157 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.379248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.379368 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.379488 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.379609 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.379759 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.379886 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.380019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81495) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.380098 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.380261 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.380414 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81494) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 30 00:42:12.396473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.396764 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 30 00:42:12.397199 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.397321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.397465 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 30 00:42:12.397606 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 30 00:42:12.397732 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.397852 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.397932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.398048 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 30 00:42:12.398261 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.398405 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.398486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.398677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.399010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.399274 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.408020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.408296 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.408564 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.408686 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.408823 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.408926 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.409121 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.409402 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.415780 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81495) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.416436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81494) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.418656 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81495) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.418946 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.419228 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.419350 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81494) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.419478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.419561 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.419781 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.419876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.420077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.420204 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.420395 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.420541 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.420676 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.420756 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.420890 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.421013 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.421159 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.421241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.421516 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.421640 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.421775 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.422046 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.444171 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.445044 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.445618 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81495) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.445902 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.446527 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81494) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.446808 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.452288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81495) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.452573 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.452856 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.453189 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81494) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.453482 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.453756 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.459373 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81495) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.459658 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.460396 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81494) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.460686 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.463828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81495) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.464116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.464411 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.464688 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.464812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81494) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.464958 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.465159 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.465283 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.465409 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.465491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.465675 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.465799 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.465920 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.466234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.466499 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.466760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.481185 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.481460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.482246 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.482524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.485308 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81495) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.486381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81494) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.488526 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.489410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.489567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.489830 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.494103 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81495) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.494372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81494) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.497275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81495) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.497572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81494) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.500530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.500800 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.500937 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.501094 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.501218 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.501408 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.501586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.501712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.501827 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.501974 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.502128 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.502256 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.517779 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81495) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.517947 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81494) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.518083 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.518235 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.518353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.518483 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.518597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.518710 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.525413 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.525582 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.526942 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.527166 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.528523 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.528699 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.528822 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.528938 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.531402 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81495) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.531570 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81494) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.531696 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.531830 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.532107 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.532294 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.532457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.532575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.532702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.532818 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.543711 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81495) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.543903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81494) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.544026 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.544142 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.544393 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.544583 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.544709 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.544817 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.544929 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.545060 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.545216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.545342 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.545496 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.545620 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.560770 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81494) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.561038 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81495) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.563983 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81494) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.564230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81495) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.566976 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.567145 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.568442 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.568716 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.570079 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81494) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.570325 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81495) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.570494 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.570617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.570788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.570905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.576919 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.577144 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.578405 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81494) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.578612 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81495) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.581387 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.581640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.581810 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.581935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.582051 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.582183 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.582307 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.582421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.590004 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81494) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.590290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.590475 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81495) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.590616 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.590734 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.590848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.598533 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81495) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.598719 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81494) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 30 00:42:12.598848 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.598963 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.599070 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.599183 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.599297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.599418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.599576 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.599747 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.600015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.600142 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.600271 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.600394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.600608 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.600777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.600915 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.601035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.601150 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.601264 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.601378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.601540 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.601657 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.601770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.601885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.601998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.602111 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81494) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.602233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81495) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 30 00:42:12.635029 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55d66c5f99c0 pid: 81494 (default app) Oct 30 00:42:12.635155 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55d66c5f99c0 pid: 81495 (default app) Oct 30 00:42:12.638321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3433cdc-c529-4829-aed9-5f008fb5b78f None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:12.638608 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3433cdc-c529-4829-aed9-5f008fb5b78f None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:12.638931 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3433cdc-c529-4829-aed9-5f008fb5b78f None None] PATH_INFO: `/v3` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:12.643064 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1/1] 10.222.0.16 () {50 vars in 791 bytes} [Sun Oct 30 00:42:12 2022] GET /identity/v3/ => generated 255 bytes in 7 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:19.396057 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b794ba-e365-4f1c-a402-9061286ec8e5 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:19.396911 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b794ba-e365-4f1c-a402-9061286ec8e5 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:19.397295 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b794ba-e365-4f1c-a402-9061286ec8e5 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:19.404395 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1/2] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:19 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:19.412953 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ae7b4d-0c3b-4e3a-9f1b-2136912f1be7 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:19.413487 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ae7b4d-0c3b-4e3a-9f1b-2136912f1be7 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:19.413487 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ae7b4d-0c3b-4e3a-9f1b-2136912f1be7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:19.491385 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: INFO dbcounter [None req-d8ae7b4d-0c3b-4e3a-9f1b-2136912f1be7 None None] Registered counter for database keystone Oct 30 00:42:19.512583 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG oslo_db.sqlalchemy.engines [None req-d8ae7b4d-0c3b-4e3a-9f1b-2136912f1be7 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81494) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:326}} Oct 30 00:42:19.524397 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writer thread running {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 30 00:42:19.713366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG passlib.handlers.bcrypt [None req-d8ae7b4d-0c3b-4e3a-9f1b-2136912f1be7 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81494) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 30 00:42:19.713650 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG passlib.handlers.bcrypt [None req-d8ae7b4d-0c3b-4e3a-9f1b-2136912f1be7 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81494) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 30 00:42:19.772217 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-d8ae7b4d-0c3b-4e3a-9f1b-2136912f1be7 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:19.773799 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d8ae7b4d-0c3b-4e3a-9f1b-2136912f1be7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:19.919249 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 2/3] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:19 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 508 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 30 00:42:20.427495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b794ba-e365-4f1c-a402-9061286ec8e5 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:20.497262 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: INFO dbcounter [None req-24b794ba-e365-4f1c-a402-9061286ec8e5 None None] Registered counter for database keystone Oct 30 00:42:20.522623 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG oslo_db.sqlalchemy.engines [None req-24b794ba-e365-4f1c-a402-9061286ec8e5 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81495) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:326}} Oct 30 00:42:20.534067 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writer thread running {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 30 00:42:20.660447 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b794ba-e365-4f1c-a402-9061286ec8e5 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:20.665839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b09ddf7-e608-4592-8fb9-fcc62c3187d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:20.666460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b09ddf7-e608-4592-8fb9-fcc62c3187d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:20.666715 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b09ddf7-e608-4592-8fb9-fcc62c3187d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:20.666986 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b09ddf7-e608-4592-8fb9-fcc62c3187d1 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:20.676119 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b09ddf7-e608-4592-8fb9-fcc62c3187d1 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:20.727044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: WARNING keystone.common.rbac_enforcer.enforcer [None req-6b09ddf7-e608-4592-8fb9-fcc62c3187d1 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 30 00:42:20.727386 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b09ddf7-e608-4592-8fb9-fcc62c3187d1 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:20.737639 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 2/4] 10.222.0.16 () {58 vars in 1324 bytes} [Sun Oct 30 00:42:20 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 30 00:42:20.745117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ae7b4d-0c3b-4e3a-9f1b-2136912f1be7 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:20.767462 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ae7b4d-0c3b-4e3a-9f1b-2136912f1be7 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:20.774143 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4276c07c-95c1-4514-8078-e5832bbf1804 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:20.775254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4276c07c-95c1-4514-8078-e5832bbf1804 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:20.775533 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4276c07c-95c1-4514-8078-e5832bbf1804 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:20.775856 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4276c07c-95c1-4514-8078-e5832bbf1804 admin admin] PATH_INFO: `/v3/projects/3907972f9f6248debeeda8ecd254820e` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:20.779870 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4276c07c-95c1-4514-8078-e5832bbf1804 admin admin] RBAC: Authorizing `identity:get_project(project_id=3907972f9f6248debeeda8ecd254820e)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:20.829450 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: WARNING keystone.common.rbac_enforcer.enforcer [None req-4276c07c-95c1-4514-8078-e5832bbf1804 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 30 00:42:20.829786 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4276c07c-95c1-4514-8078-e5832bbf1804 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:20.834447 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 3/5] 10.222.0.16 () {56 vars in 1207 bytes} [Sun Oct 30 00:42:20 2022] GET /identity/v3/projects/3907972f9f6248debeeda8ecd254820e => generated 344 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:22.963869 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d433c5-69e3-4b4f-b09c-00f76ccceef5 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:22.964432 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d433c5-69e3-4b4f-b09c-00f76ccceef5 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:22.964432 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d433c5-69e3-4b4f-b09c-00f76ccceef5 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:22.968446 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 3/6] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:22 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:42:22.975282 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1746855c-8056-4034-900d-82df297ca8f7 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:22.976020 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1746855c-8056-4034-900d-82df297ca8f7 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:22.976168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1746855c-8056-4034-900d-82df297ca8f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:23.002314 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-1746855c-8056-4034-900d-82df297ca8f7 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:23.003877 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-1746855c-8056-4034-900d-82df297ca8f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:23.013394 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 4/7] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:22 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 30 00:42:23.280204 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d433c5-69e3-4b4f-b09c-00f76ccceef5 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:23.294520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d433c5-69e3-4b4f-b09c-00f76ccceef5 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:23.299887 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea93f8cb-c06d-445e-93fe-e08fc32523bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:23.300738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea93f8cb-c06d-445e-93fe-e08fc32523bd admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:23.301006 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea93f8cb-c06d-445e-93fe-e08fc32523bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:23.301273 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea93f8cb-c06d-445e-93fe-e08fc32523bd admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:23.314959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea93f8cb-c06d-445e-93fe-e08fc32523bd admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:23.318023 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea93f8cb-c06d-445e-93fe-e08fc32523bd admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:23.327757 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 4/8] 10.222.0.16 () {58 vars in 1324 bytes} [Sun Oct 30 00:42:23 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 30 00:42:23.333360 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1746855c-8056-4034-900d-82df297ca8f7 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:23.342806 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1746855c-8056-4034-900d-82df297ca8f7 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:23.347815 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534084db-217d-4abc-b990-610dfd3b8b5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:23.348453 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534084db-217d-4abc-b990-610dfd3b8b5f admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:23.348735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534084db-217d-4abc-b990-610dfd3b8b5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:23.349142 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534084db-217d-4abc-b990-610dfd3b8b5f admin admin] PATH_INFO: `/v3/users/ca6c0649e64b4d38bdd14a04f31beb73` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:23.350538 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534084db-217d-4abc-b990-610dfd3b8b5f admin admin] RBAC: Authorizing `identity:get_user(user_id=ca6c0649e64b4d38bdd14a04f31beb73)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:23.352396 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534084db-217d-4abc-b990-610dfd3b8b5f admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:23.353929 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 5/9] 10.222.0.16 () {56 vars in 1198 bytes} [Sun Oct 30 00:42:23 2022] GET /identity/v3/users/ca6c0649e64b4d38bdd14a04f31beb73 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:25.528675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ddb2cd-c143-4734-b886-1c2504e454e4 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:25.529268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ddb2cd-c143-4734-b886-1c2504e454e4 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:25.529268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ddb2cd-c143-4734-b886-1c2504e454e4 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:25.530529 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 5/10] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:25.535991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb526b8-0754-4261-93d9-f3a0859a12a6 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:25.536509 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb526b8-0754-4261-93d9-f3a0859a12a6 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:25.536656 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb526b8-0754-4261-93d9-f3a0859a12a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:25.556250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-9eb526b8-0754-4261-93d9-f3a0859a12a6 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:25.557470 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-9eb526b8-0754-4261-93d9-f3a0859a12a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:25.568472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114813f7-c331-415f-a81a-76a5c630a94b None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:25.568773 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114813f7-c331-415f-a81a-76a5c630a94b None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:25.569046 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114813f7-c331-415f-a81a-76a5c630a94b None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:25.570371 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 6/11] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:25.575465 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02492b34-a271-4154-a724-ae8cb9f649cb None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:25.575734 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02492b34-a271-4154-a724-ae8cb9f649cb None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:25.576047 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02492b34-a271-4154-a724-ae8cb9f649cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:25.618093 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG passlib.handlers.bcrypt [None req-02492b34-a271-4154-a724-ae8cb9f649cb None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81495) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 30 00:42:25.618403 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG passlib.handlers.bcrypt [None req-02492b34-a271-4154-a724-ae8cb9f649cb None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81495) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 30 00:42:25.642900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 6/12] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:42:25.666980 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-02492b34-a271-4154-a724-ae8cb9f649cb None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:25.668565 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-02492b34-a271-4154-a724-ae8cb9f649cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:25.767761 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 7/13] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:42:25.906999 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb526b8-0754-4261-93d9-f3a0859a12a6 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:25.931860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:26.014404 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb526b8-0754-4261-93d9-f3a0859a12a6 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:26.026133 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02492b34-a271-4154-a724-ae8cb9f649cb None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:26.052518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9477366-77b8-4dad-a08b-dd2ef97d3564 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:26.053164 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9477366-77b8-4dad-a08b-dd2ef97d3564 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:26.053436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9477366-77b8-4dad-a08b-dd2ef97d3564 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:26.053701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9477366-77b8-4dad-a08b-dd2ef97d3564 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:26.065420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9477366-77b8-4dad-a08b-dd2ef97d3564 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:42:26.074278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-a9477366-77b8-4dad-a08b-dd2ef97d3564 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:42:26.074278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:26.074278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:42:26.074278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:42:26.074278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:42:26.074278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:42:26.074278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:42:26.074278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:42:26.074278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:42:26.074278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:42:26.074278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:42:26.074278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:42:26.074278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:42:26.074278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:42:26.074278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:42:26.074278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:42:26.075179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:42:26.075179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:42:26.075179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:42:26.075179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:42:26.075179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:42:26.075179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:42:26.075179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:42:26.075179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:42:26.075179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:42:26.075179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:42:26.075179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:42:26.075179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:42:26.075179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:42:26.075179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:42:26.075179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:42:26.075179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:42:26.076115 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:42:26.076115 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:42:26.076115 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:42:26.076115 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:42:26.076115 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:42:26.076115 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:42:26.076115 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:42:26.076115 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:42:26.076115 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:42:26.076115 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:42:26.076115 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:42:26.076115 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:42:26.076115 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 7/14] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:42:25 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:42:26.081471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9477366-77b8-4dad-a08b-dd2ef97d3564 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:26.130865 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02492b34-a271-4154-a724-ae8cb9f649cb None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:26.133321 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:26.161723 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:26.166790 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e52ea6-affb-40e7-8873-793fd4819c15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:26.167664 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e52ea6-affb-40e7-8873-793fd4819c15 None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:26.167969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e52ea6-affb-40e7-8873-793fd4819c15 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:26.168257 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e52ea6-affb-40e7-8873-793fd4819c15 None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:26.169228 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e52ea6-affb-40e7-8873-793fd4819c15 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:26.170661 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e52ea6-affb-40e7-8873-793fd4819c15 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:26.180352 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9477366-77b8-4dad-a08b-dd2ef97d3564 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:26.181906 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.sql.core [None req-88e52ea6-affb-40e7-8873-793fd4819c15 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 30 00:42:26.181906 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 30 00:42:26.181906 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [parameters: {'id': '3adbf742abce416b9acb1cd807a6eaee', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 30 00:42:26.181906 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81495) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 30 00:42:26.200699 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-88e52ea6-affb-40e7-8873-793fd4819c15 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 30 00:42:26.200699 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:26.200699 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 30 00:42:26.200699 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 30 00:42:26.200699 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 30 00:42:26.200699 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 30 00:42:26.200699 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 30 00:42:26.200699 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application result = self._query(query) Oct 30 00:42:26.200699 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 30 00:42:26.200699 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application conn.query(q) Oct 30 00:42:26.200699 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 30 00:42:26.200699 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 30 00:42:26.200699 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 30 00:42:26.200699 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application result.read() Oct 30 00:42:26.201756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 30 00:42:26.201756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 30 00:42:26.201756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 30 00:42:26.201756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 30 00:42:26.201756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 30 00:42:26.201756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 30 00:42:26.201756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 30 00:42:26.201756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 30 00:42:26.201756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 30 00:42:26.201756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:42:26.201756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 30 00:42:26.201756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:42:26.201756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:26.201756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 30 00:42:26.201756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 30 00:42:26.201756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 30 00:42:26.201756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 30 00:42:26.201756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 30 00:42:26.202850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application next(self.gen) Oct 30 00:42:26.202850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 30 00:42:26.202850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application yield resource Oct 30 00:42:26.202850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 30 00:42:26.202850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application next(self.gen) Oct 30 00:42:26.202850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 30 00:42:26.202850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self.session.rollback() Oct 30 00:42:26.202850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 30 00:42:26.202850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self.force_reraise() Oct 30 00:42:26.202850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 30 00:42:26.202850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise self.value Oct 30 00:42:26.202850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 30 00:42:26.202850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 30 00:42:26.202850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 30 00:42:26.202850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application session.commit() Oct 30 00:42:26.202850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 30 00:42:26.202850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 30 00:42:26.203959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 30 00:42:26.203959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._prepare_impl() Oct 30 00:42:26.203959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 30 00:42:26.203959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self.session.flush() Oct 30 00:42:26.203959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 30 00:42:26.203959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._flush(objects) Oct 30 00:42:26.203959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 30 00:42:26.203959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 30 00:42:26.203959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 30 00:42:26.203959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application compat.raise_( Oct 30 00:42:26.203959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 30 00:42:26.203959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception Oct 30 00:42:26.203959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 30 00:42:26.203959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application flush_context.execute() Oct 30 00:42:26.203959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 30 00:42:26.203959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rec.execute(self) Oct 30 00:42:26.203959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 30 00:42:26.204998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 30 00:42:26.204998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 30 00:42:26.204998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 30 00:42:26.204998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 30 00:42:26.204998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 30 00:42:26.204998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 30 00:42:26.204998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 30 00:42:26.204998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 30 00:42:26.204998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 30 00:42:26.204998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 30 00:42:26.204998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 30 00:42:26.204998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 30 00:42:26.204998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 30 00:42:26.204998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 30 00:42:26.204998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 30 00:42:26.205904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 30 00:42:26.205904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception Oct 30 00:42:26.205904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 30 00:42:26.205904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 30 00:42:26.205904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 30 00:42:26.205904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 30 00:42:26.205904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 30 00:42:26.205904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application result = self._query(query) Oct 30 00:42:26.205904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 30 00:42:26.205904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application conn.query(q) Oct 30 00:42:26.205904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 30 00:42:26.205904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 30 00:42:26.205904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 30 00:42:26.205904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application result.read() Oct 30 00:42:26.205904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 30 00:42:26.205904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 30 00:42:26.205904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 30 00:42:26.206990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 30 00:42:26.206990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 30 00:42:26.206990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 30 00:42:26.206990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 30 00:42:26.206990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 30 00:42:26.206990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 30 00:42:26.206990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 30 00:42:26.206990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [parameters: {'id': '3adbf742abce416b9acb1cd807a6eaee', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 30 00:42:26.206990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 30 00:42:26.206990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:42:26.206990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:42:26.206990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:42:26.206990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:26.206990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 30 00:42:26.206990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 30 00:42:26.207951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 30 00:42:26.207951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 30 00:42:26.207951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 30 00:42:26.207951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:42:26.207951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:42:26.207951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:42:26.207951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:26.207951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:42:26.207951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:42:26.207951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:42:26.207951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:42:26.207951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:42:26.207951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:42:26.207951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:42:26.207951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:42:26.207951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:42:26.207951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:42:26.209233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 30 00:42:26.209233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 30 00:42:26.209233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:42:26.209233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:42:26.209233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 30 00:42:26.209233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 30 00:42:26.209233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:42:26.209233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:42:26.209233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 30 00:42:26.209233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 30 00:42:26.209233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 30 00:42:26.209233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:42:26.209233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 8/15] 10.222.0.16 () {60 vars in 1137 bytes} [Sun Oct 30 00:42:26 2022] POST /identity/v3/domains => generated 190 bytes in 176 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:42:26.209233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e52ea6-affb-40e7-8873-793fd4819c15 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:26.225269 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48292da3-69e4-4f96-8760-3918ef6e6732 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:26.225906 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48292da3-69e4-4f96-8760-3918ef6e6732 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:26.226182 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48292da3-69e4-4f96-8760-3918ef6e6732 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:26.226449 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48292da3-69e4-4f96-8760-3918ef6e6732 None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:26.227115 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48292da3-69e4-4f96-8760-3918ef6e6732 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:26.227410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48292da3-69e4-4f96-8760-3918ef6e6732 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:26.228819 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48292da3-69e4-4f96-8760-3918ef6e6732 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:26.240582 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 8/16] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:42:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:42:26.246304 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48292da3-69e4-4f96-8760-3918ef6e6732 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:26.307790 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e52ea6-affb-40e7-8873-793fd4819c15 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:26.341492 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f98f486-85b2-415e-bb8a-fea6655e36be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:26.344881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f98f486-85b2-415e-bb8a-fea6655e36be None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:26.345161 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f98f486-85b2-415e-bb8a-fea6655e36be None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:26.345439 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f98f486-85b2-415e-bb8a-fea6655e36be None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:26.346667 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48292da3-69e4-4f96-8760-3918ef6e6732 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:26.364000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f98f486-85b2-415e-bb8a-fea6655e36be None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:26.365559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f98f486-85b2-415e-bb8a-fea6655e36be None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:26.381269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-4f98f486-85b2-415e-bb8a-fea6655e36be None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:42:26.381269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:26.381269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:42:26.381269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:42:26.381269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:42:26.381269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:42:26.381269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:42:26.381269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:42:26.381269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:42:26.381269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:42:26.381269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:42:26.381269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:42:26.381269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:42:26.381269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:42:26.381269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:42:26.381269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:42:26.382148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:42:26.382148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:42:26.382148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:42:26.382148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:42:26.382148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:42:26.382148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:42:26.382148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:42:26.382148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:42:26.382148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:42:26.382148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:42:26.382148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:42:26.382148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:42:26.382148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:42:26.382148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:42:26.382148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:42:26.382148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:42:26.382148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:42:26.382940 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:42:26.382940 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:42:26.382940 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:42:26.383094 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8b4933-b3d7-4908-98a1-ce54965b987a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:26.383176 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 9/17] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:42:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:42:26.383239 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8b4933-b3d7-4908-98a1-ce54965b987a None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:26.383239 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8b4933-b3d7-4908-98a1-ce54965b987a None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:26.383239 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8b4933-b3d7-4908-98a1-ce54965b987a None admin] PATH_INFO: `/v3/users/ca6c0649e64b4d38bdd14a04f31beb73` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:26.384497 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8b4933-b3d7-4908-98a1-ce54965b987a None admin] RBAC: Authorizing `identity:get_user(user_id=ca6c0649e64b4d38bdd14a04f31beb73)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:26.385931 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8b4933-b3d7-4908-98a1-ce54965b987a None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:26.387529 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 9/18] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:42:26 2022] GET /identity/v3/users/ca6c0649e64b4d38bdd14a04f31beb73 => generated 247 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:26.388237 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f98f486-85b2-415e-bb8a-fea6655e36be None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:26.392684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8b4933-b3d7-4908-98a1-ce54965b987a None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:26.407326 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:26.413517 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:26.490893 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f98f486-85b2-415e-bb8a-fea6655e36be None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:26.492250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8b4933-b3d7-4908-98a1-ce54965b987a None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:26.524857 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f39f9b9-7c38-423b-b32e-bbdd486b2fa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:26.525473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f39f9b9-7c38-423b-b32e-bbdd486b2fa2 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:26.525753 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f39f9b9-7c38-423b-b32e-bbdd486b2fa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:26.526022 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f39f9b9-7c38-423b-b32e-bbdd486b2fa2 None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:26.526659 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f39f9b9-7c38-423b-b32e-bbdd486b2fa2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:26.526957 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f39f9b9-7c38-423b-b32e-bbdd486b2fa2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:26.527043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876492e7-5408-4a24-9877-711712060e25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:26.527661 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876492e7-5408-4a24-9877-711712060e25 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:26.527963 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876492e7-5408-4a24-9877-711712060e25 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:26.528228 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876492e7-5408-4a24-9877-711712060e25 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:26.528377 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f39f9b9-7c38-423b-b32e-bbdd486b2fa2 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:26.529339 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876492e7-5408-4a24-9877-711712060e25 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:26.530754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876492e7-5408-4a24-9877-711712060e25 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:26.532322 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 10/19] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:42:26 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:42:26.538118 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876492e7-5408-4a24-9877-711712060e25 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:26.546124 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 10/20] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:42:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:42:26.625352 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:26.647428 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876492e7-5408-4a24-9877-711712060e25 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:26.684184 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cfee9c-204d-434a-ae21-4bf87b573956 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:26.684764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cfee9c-204d-434a-ae21-4bf87b573956 None admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:26.685039 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cfee9c-204d-434a-ae21-4bf87b573956 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:26.685308 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cfee9c-204d-434a-ae21-4bf87b573956 None admin] PATH_INFO: `/v3/domains/default/users/ca6c0649e64b4d38bdd14a04f31beb73/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:26.687614 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6cfee9c-204d-434a-ae21-4bf87b573956 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=ca6c0649e64b4d38bdd14a04f31beb73, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:26.689105 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6cfee9c-204d-434a-ae21-4bf87b573956 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:26.802894 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 11/21] 10.222.0.16 () {58 vars in 1361 bytes} [Sun Oct 30 00:42:26 2022] PUT /identity/v3/domains/default/users/ca6c0649e64b4d38bdd14a04f31beb73/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 00:42:29.178203 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2bf8b9-4ac3-45ca-aa6d-969d52b73cd9 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:29.178750 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2bf8b9-4ac3-45ca-aa6d-969d52b73cd9 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:29.178750 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2bf8b9-4ac3-45ca-aa6d-969d52b73cd9 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:29.180113 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 11/22] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:29.198552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c560d8-f39b-4ec8-bece-262c613c17e9 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:29.199070 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c560d8-f39b-4ec8-bece-262c613c17e9 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:29.199070 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c560d8-f39b-4ec8-bece-262c613c17e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:29.223091 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-15c560d8-f39b-4ec8-bece-262c613c17e9 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:29.224820 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-15c560d8-f39b-4ec8-bece-262c613c17e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:29.309502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 12/23] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:42:29.364700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc32010-dfc3-4450-bbc0-a9f2830383a0 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:29.364700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc32010-dfc3-4450-bbc0-a9f2830383a0 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:29.364700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc32010-dfc3-4450-bbc0-a9f2830383a0 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:29.365438 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 12/24] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:29.375216 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ee5c89-7cd7-43f8-bcde-2870b3496845 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:29.375516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ee5c89-7cd7-43f8-bcde-2870b3496845 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:29.375793 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ee5c89-7cd7-43f8-bcde-2870b3496845 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:29.398424 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-42ee5c89-7cd7-43f8-bcde-2870b3496845 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:29.399628 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-42ee5c89-7cd7-43f8-bcde-2870b3496845 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:29.423440 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7a01cf-e1ad-4dd6-8bf4-966b62d2b076 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:29.424184 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7a01cf-e1ad-4dd6-8bf4-966b62d2b076 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:29.424642 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7a01cf-e1ad-4dd6-8bf4-966b62d2b076 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:29.426138 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 13/25] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:42:29.431558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7749d2c-4a73-4240-878b-8d27efdf313a None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:29.432107 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7749d2c-4a73-4240-878b-8d27efdf313a None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:29.432559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7749d2c-4a73-4240-878b-8d27efdf313a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:29.457832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-d7749d2c-4a73-4240-878b-8d27efdf313a None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:29.459351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-d7749d2c-4a73-4240-878b-8d27efdf313a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:29.492205 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:29.498892 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 13/26] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:42:29.502939 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8871ed0-5678-4412-a4eb-7d2a0f35d405 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:29.503436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8871ed0-5678-4412-a4eb-7d2a0f35d405 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:29.504022 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8871ed0-5678-4412-a4eb-7d2a0f35d405 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:29.508547 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 14/27] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:29 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:29.514098 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6d3df4-f71a-4c64-92c2-2a14b82b3628 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:29.515914 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6d3df4-f71a-4c64-92c2-2a14b82b3628 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:29.516378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6d3df4-f71a-4c64-92c2-2a14b82b3628 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:29.549398 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-4a6d3df4-f71a-4c64-92c2-2a14b82b3628 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:29.552597 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-4a6d3df4-f71a-4c64-92c2-2a14b82b3628 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:29.586664 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 14/28] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:42:29.643260 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 15/29] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:42:29.651974 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7749d2c-4a73-4240-878b-8d27efdf313a None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:29.665207 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:29.766137 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7749d2c-4a73-4240-878b-8d27efdf313a None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:29.806420 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b465068-8763-4d72-b900-9a00c349d762 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:29.807451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b465068-8763-4d72-b900-9a00c349d762 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:29.807957 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b465068-8763-4d72-b900-9a00c349d762 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:29.808412 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b465068-8763-4d72-b900-9a00c349d762 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:29.818741 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b465068-8763-4d72-b900-9a00c349d762 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:29.820443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b465068-8763-4d72-b900-9a00c349d762 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:29.835287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-2b465068-8763-4d72-b900-9a00c349d762 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:42:29.835287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:29.835287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:42:29.835287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:42:29.835287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:42:29.835287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:42:29.835287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:42:29.835287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:42:29.835287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:42:29.835287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:42:29.835287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:42:29.835287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:42:29.835287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:42:29.835287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:42:29.835287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:42:29.835287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:42:29.836226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:42:29.836226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:42:29.836226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:42:29.836226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:42:29.836226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:42:29.836226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:42:29.836226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:42:29.836226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:42:29.836226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:42:29.836226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:42:29.836226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:42:29.836226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:42:29.836226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:42:29.836226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:42:29.836226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:42:29.836226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:42:29.836226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:42:29.837008 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:42:29.837008 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:42:29.837008 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:42:29.838778 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 15/30] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:42:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:42:29.844561 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6d3df4-f71a-4c64-92c2-2a14b82b3628 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:29.859281 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b465068-8763-4d72-b900-9a00c349d762 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:29.975588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6d3df4-f71a-4c64-92c2-2a14b82b3628 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:29.992146 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:30.036542 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:30.039394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b465068-8763-4d72-b900-9a00c349d762 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:30.047001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39eca92-b7c6-42a5-9003-0ff864e26b94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:30.047631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39eca92-b7c6-42a5-9003-0ff864e26b94 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:30.047943 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39eca92-b7c6-42a5-9003-0ff864e26b94 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:30.048492 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39eca92-b7c6-42a5-9003-0ff864e26b94 None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:30.048878 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39eca92-b7c6-42a5-9003-0ff864e26b94 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:30.049169 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39eca92-b7c6-42a5-9003-0ff864e26b94 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:30.050688 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39eca92-b7c6-42a5-9003-0ff864e26b94 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:30.078262 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 16/31] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:42:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:42:30.079090 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39eca92-b7c6-42a5-9003-0ff864e26b94 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:30.079327 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a1ce1e-1397-4417-92eb-96f462e73a46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:30.080319 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a1ce1e-1397-4417-92eb-96f462e73a46 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:30.080831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a1ce1e-1397-4417-92eb-96f462e73a46 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:30.081300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a1ce1e-1397-4417-92eb-96f462e73a46 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:30.093724 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a1ce1e-1397-4417-92eb-96f462e73a46 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:42:30.102081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-72a1ce1e-1397-4417-92eb-96f462e73a46 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:42:30.102081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:30.102081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:42:30.102081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:42:30.102081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:42:30.102081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:42:30.102081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:42:30.102081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:42:30.102081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:42:30.102081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:42:30.102081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:42:30.102081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:42:30.102081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:42:30.102081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:42:30.102081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:42:30.102081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:42:30.102932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:42:30.102932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:42:30.102932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:42:30.102932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:42:30.102932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:42:30.102932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:42:30.102932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:42:30.102932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:42:30.102932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:42:30.102932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:42:30.102932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:42:30.102932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:42:30.102932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:42:30.102932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:42:30.102932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:42:30.102932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:42:30.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:42:30.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:42:30.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:42:30.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:42:30.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:42:30.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:42:30.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:42:30.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:42:30.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:42:30.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:42:30.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:42:30.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:42:30.106211 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 16/32] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:42:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:42:30.107597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a1ce1e-1397-4417-92eb-96f462e73a46 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:30.183238 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39eca92-b7c6-42a5-9003-0ff864e26b94 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:30.219229 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe77d501-eb91-464f-ab6e-0cc27a71a829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:30.220267 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe77d501-eb91-464f-ab6e-0cc27a71a829 None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:30.220382 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe77d501-eb91-464f-ab6e-0cc27a71a829 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:30.220741 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe77d501-eb91-464f-ab6e-0cc27a71a829 None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:30.221626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe77d501-eb91-464f-ab6e-0cc27a71a829 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:30.223072 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe77d501-eb91-464f-ab6e-0cc27a71a829 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:30.234219 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a1ce1e-1397-4417-92eb-96f462e73a46 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:30.247388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 17/33] 10.222.0.16 () {60 vars in 1131 bytes} [Sun Oct 30 00:42:30 2022] POST /identity/v3/roles => generated 225 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:42:30.248750 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe77d501-eb91-464f-ab6e-0cc27a71a829 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:30.260743 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:30.274774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15cd9827-4ec8-45df-bc1c-58c11f251d8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:30.275426 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15cd9827-4ec8-45df-bc1c-58c11f251d8b None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:30.275705 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15cd9827-4ec8-45df-bc1c-58c11f251d8b None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:30.276022 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15cd9827-4ec8-45df-bc1c-58c11f251d8b None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:30.276747 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15cd9827-4ec8-45df-bc1c-58c11f251d8b None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:30.278494 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15cd9827-4ec8-45df-bc1c-58c11f251d8b None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:30.315856 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:30.377190 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe77d501-eb91-464f-ab6e-0cc27a71a829 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:30.415267 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f772db16-d6dd-401f-8b17-aede78843ce0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:30.416094 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f772db16-d6dd-401f-8b17-aede78843ce0 None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:30.416464 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f772db16-d6dd-401f-8b17-aede78843ce0 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:30.416829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f772db16-d6dd-401f-8b17-aede78843ce0 None admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:30.417569 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f772db16-d6dd-401f-8b17-aede78843ce0 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:30.419225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f772db16-d6dd-401f-8b17-aede78843ce0 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:30.423413 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 17/34] 10.222.0.16 () {60 vars in 1131 bytes} [Sun Oct 30 00:42:30 2022] POST /identity/v3/roles => generated 219 bytes in 316 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:42:30.424885 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15cd9827-4ec8-45df-bc1c-58c11f251d8b None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:30.458671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 18/35] 10.222.0.16 () {60 vars in 1141 bytes} [Sun Oct 30 00:42:30 2022] POST /identity/v3/projects => generated 301 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:42:30.566892 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15cd9827-4ec8-45df-bc1c-58c11f251d8b None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:30.618876 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64866e98-6157-4f86-8579-59a3eee0ac05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:30.619867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64866e98-6157-4f86-8579-59a3eee0ac05 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:30.620305 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64866e98-6157-4f86-8579-59a3eee0ac05 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:30.620735 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64866e98-6157-4f86-8579-59a3eee0ac05 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:30.621603 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64866e98-6157-4f86-8579-59a3eee0ac05 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:30.622050 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64866e98-6157-4f86-8579-59a3eee0ac05 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:30.623876 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64866e98-6157-4f86-8579-59a3eee0ac05 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:30.643924 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 18/36] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:42:30 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:30.649968 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f772db16-d6dd-401f-8b17-aede78843ce0 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:30.783832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f772db16-d6dd-401f-8b17-aede78843ce0 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:30.785847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:30.822946 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f81673-ed9f-43ba-9b57-59e2368076ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:30.823505 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f81673-ed9f-43ba-9b57-59e2368076ab None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:30.823826 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f81673-ed9f-43ba-9b57-59e2368076ab None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:30.824105 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f81673-ed9f-43ba-9b57-59e2368076ab None admin] PATH_INFO: `/v3/users/ca6c0649e64b4d38bdd14a04f31beb73` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:30.825253 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f81673-ed9f-43ba-9b57-59e2368076ab None admin] RBAC: Authorizing `identity:get_user(user_id=ca6c0649e64b4d38bdd14a04f31beb73)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:30.826725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f81673-ed9f-43ba-9b57-59e2368076ab None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:30.828935 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 19/37] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:42:30 2022] GET /identity/v3/users/ca6c0649e64b4d38bdd14a04f31beb73 => generated 247 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:30.834472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64866e98-6157-4f86-8579-59a3eee0ac05 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:30.889197 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:30.947117 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64866e98-6157-4f86-8579-59a3eee0ac05 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:30.966044 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a934959-2997-4bb6-acdf-a6a394cf652d None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:30.966333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a934959-2997-4bb6-acdf-a6a394cf652d None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:30.966603 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a934959-2997-4bb6-acdf-a6a394cf652d None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:30.968111 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 20/38] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:42:30.972781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0190a6-efd1-45c4-8497-2853205141af None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:30.973067 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0190a6-efd1-45c4-8497-2853205141af None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:30.973343 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0190a6-efd1-45c4-8497-2853205141af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:30.983892 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14323d7-2618-4439-92a4-7fbd89552148 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:30.984554 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14323d7-2618-4439-92a4-7fbd89552148 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:30.984831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14323d7-2618-4439-92a4-7fbd89552148 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:30.985100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14323d7-2618-4439-92a4-7fbd89552148 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:30.986274 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14323d7-2618-4439-92a4-7fbd89552148 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:30.988163 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14323d7-2618-4439-92a4-7fbd89552148 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:30.990268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 19/39] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:42:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:30.992489 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-ee0190a6-efd1-45c4-8497-2853205141af None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:30.993682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-ee0190a6-efd1-45c4-8497-2853205141af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:30.995756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14323d7-2618-4439-92a4-7fbd89552148 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:31.110273 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14323d7-2618-4439-92a4-7fbd89552148 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:31.125857 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:31.149265 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64436f13-7152-46e4-817e-2ca521349338 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:31.149899 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64436f13-7152-46e4-817e-2ca521349338 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:31.150159 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64436f13-7152-46e4-817e-2ca521349338 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:31.150431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64436f13-7152-46e4-817e-2ca521349338 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:31.151157 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64436f13-7152-46e4-817e-2ca521349338 None admin] RBAC: Adding query filter params (role.id=35f59686b5d84291a8d9baf19287eb5a, scope.domain.id=default, user.id=ca6c0649e64b4d38bdd14a04f31beb73) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:31.151647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64436f13-7152-46e4-817e-2ca521349338 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:31.153787 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64436f13-7152-46e4-817e-2ca521349338 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:31.171758 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 20/40] 10.222.0.16 () {56 vars in 1322 bytes} [Sun Oct 30 00:42:30 2022] GET /identity/v3/role_assignments?user.id=ca6c0649e64b4d38bdd14a04f31beb73&scope.domain.id=default&role.id=35f59686b5d84291a8d9baf19287eb5a => generated 530 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:42:31.181926 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 21/41] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:42:31.374926 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fead99-83a5-47ba-b779-06ee2a2f287d None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:31.375398 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fead99-83a5-47ba-b779-06ee2a2f287d None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:31.375398 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fead99-83a5-47ba-b779-06ee2a2f287d None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:31.376813 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 21/42] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:31.382212 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96397a98-b8a7-481a-9276-5505a9ad8cc5 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:31.382605 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96397a98-b8a7-481a-9276-5505a9ad8cc5 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:31.382771 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96397a98-b8a7-481a-9276-5505a9ad8cc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:31.402072 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-96397a98-b8a7-481a-9276-5505a9ad8cc5 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:31.403299 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-96397a98-b8a7-481a-9276-5505a9ad8cc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:31.447170 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fead99-83a5-47ba-b779-06ee2a2f287d None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:31.470716 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 22/43] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:42:31.551308 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fead99-83a5-47ba-b779-06ee2a2f287d None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:31.584604 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4431a2e2-a471-4cbe-9a00-60736bf04278 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:31.585193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4431a2e2-a471-4cbe-9a00-60736bf04278 None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:31.585465 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4431a2e2-a471-4cbe-9a00-60736bf04278 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:31.585733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4431a2e2-a471-4cbe-9a00-60736bf04278 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:31.586385 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4431a2e2-a471-4cbe-9a00-60736bf04278 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:31.587776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4431a2e2-a471-4cbe-9a00-60736bf04278 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:31.602574 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 22/44] 10.222.0.16 () {60 vars in 1131 bytes} [Sun Oct 30 00:42:31 2022] POST /identity/v3/roles => generated 223 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:42:31.737660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96397a98-b8a7-481a-9276-5505a9ad8cc5 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:31.770918 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:31.842436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96397a98-b8a7-481a-9276-5505a9ad8cc5 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:31.876938 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38030663-366f-4cc5-a9f3-6a3a1c1be14c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:31.877562 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38030663-366f-4cc5-a9f3-6a3a1c1be14c None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:31.877836 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38030663-366f-4cc5-a9f3-6a3a1c1be14c None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:31.878100 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38030663-366f-4cc5-a9f3-6a3a1c1be14c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:31.879189 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38030663-366f-4cc5-a9f3-6a3a1c1be14c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:31.880648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38030663-366f-4cc5-a9f3-6a3a1c1be14c None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:31.882219 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 23/45] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:42:31 2022] GET /identity/v3/domains/default => generated 199 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:31.888683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4431a2e2-a471-4cbe-9a00-60736bf04278 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:31.930270 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:32.006761 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4431a2e2-a471-4cbe-9a00-60736bf04278 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:32.047066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1aff8a-4424-4e30-bf46-01525ae8437a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:32.047682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1aff8a-4424-4e30-bf46-01525ae8437a None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:32.047994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1aff8a-4424-4e30-bf46-01525ae8437a None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:32.048266 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1aff8a-4424-4e30-bf46-01525ae8437a None admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:32.048901 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1aff8a-4424-4e30-bf46-01525ae8437a None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:32.050360 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1aff8a-4424-4e30-bf46-01525ae8437a None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:32.085095 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 23/46] 10.222.0.16 () {60 vars in 1141 bytes} [Sun Oct 30 00:42:31 2022] POST /identity/v3/projects => generated 312 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:42:34.192488 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4feecbb2-fad6-4406-bcba-05dd6b52c7ea None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:34.193235 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4feecbb2-fad6-4406-bcba-05dd6b52c7ea None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:34.193235 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4feecbb2-fad6-4406-bcba-05dd6b52c7ea None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:34.194490 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 24/47] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:34.203679 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedad178-31f9-4c25-9c64-974550d9c131 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:34.204436 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedad178-31f9-4c25-9c64-974550d9c131 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:34.204568 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedad178-31f9-4c25-9c64-974550d9c131 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:34.227690 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-dedad178-31f9-4c25-9c64-974550d9c131 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:34.229250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-dedad178-31f9-4c25-9c64-974550d9c131 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:34.300941 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 24/48] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:42:34.572282 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4feecbb2-fad6-4406-bcba-05dd6b52c7ea None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:34.688492 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:34.695169 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4feecbb2-fad6-4406-bcba-05dd6b52c7ea None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:34.732889 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5282f84d-dd44-472f-8019-81252af880f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:34.733509 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5282f84d-dd44-472f-8019-81252af880f2 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:34.733787 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5282f84d-dd44-472f-8019-81252af880f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:34.734059 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5282f84d-dd44-472f-8019-81252af880f2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:34.735238 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5282f84d-dd44-472f-8019-81252af880f2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:34.736724 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5282f84d-dd44-472f-8019-81252af880f2 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:34.738262 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 25/49] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:42:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:34.744454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedad178-31f9-4c25-9c64-974550d9c131 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:34.764033 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:34.847281 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedad178-31f9-4c25-9c64-974550d9c131 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:34.882110 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eca802d-390b-41b8-93f9-48f2d05a4f21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:34.882705 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eca802d-390b-41b8-93f9-48f2d05a4f21 None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:34.882980 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eca802d-390b-41b8-93f9-48f2d05a4f21 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:34.883250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eca802d-390b-41b8-93f9-48f2d05a4f21 None admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:34.883919 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eca802d-390b-41b8-93f9-48f2d05a4f21 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:34.885368 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eca802d-390b-41b8-93f9-48f2d05a4f21 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:34.915538 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 25/50] 10.222.0.16 () {60 vars in 1140 bytes} [Sun Oct 30 00:42:34 2022] POST /identity/v3/projects => generated 298 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:42:37.043440 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d425a6a-4003-4a66-accb-1e0bb62fc2d8 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:37.044342 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d425a6a-4003-4a66-accb-1e0bb62fc2d8 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:37.044342 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d425a6a-4003-4a66-accb-1e0bb62fc2d8 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:37.046569 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 26/51] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:37.052505 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16496302-207e-4bce-9e1a-b13be9413e88 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:37.052910 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16496302-207e-4bce-9e1a-b13be9413e88 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:37.053021 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16496302-207e-4bce-9e1a-b13be9413e88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:37.073458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-16496302-207e-4bce-9e1a-b13be9413e88 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:37.074629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-16496302-207e-4bce-9e1a-b13be9413e88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:37.250856 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:37.257146 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 26/52] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:42:37.534058 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d425a6a-4003-4a66-accb-1e0bb62fc2d8 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:37.650459 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d425a6a-4003-4a66-accb-1e0bb62fc2d8 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:37.688260 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f185a2-f111-4f31-a27f-f30129271471 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:37.688877 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f185a2-f111-4f31-a27f-f30129271471 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:37.689151 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f185a2-f111-4f31-a27f-f30129271471 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:37.689419 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f185a2-f111-4f31-a27f-f30129271471 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:37.690633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f185a2-f111-4f31-a27f-f30129271471 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:37.692119 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f185a2-f111-4f31-a27f-f30129271471 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:37.693682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 27/53] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:42:37 2022] GET /identity/v3/domains/default => generated 199 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:37.699519 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16496302-207e-4bce-9e1a-b13be9413e88 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:37.800919 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16496302-207e-4bce-9e1a-b13be9413e88 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:37.836655 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c17660-1739-4dd2-9345-364ea2ed19f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:37.837250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c17660-1739-4dd2-9345-364ea2ed19f5 None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:37.837525 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c17660-1739-4dd2-9345-364ea2ed19f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:37.837791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c17660-1739-4dd2-9345-364ea2ed19f5 None admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:37.838420 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c17660-1739-4dd2-9345-364ea2ed19f5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:37.839907 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c17660-1739-4dd2-9345-364ea2ed19f5 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:37.870450 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 27/54] 10.222.0.16 () {60 vars in 1132 bytes} [Sun Oct 30 00:42:37 2022] POST /identity/v3/users => generated 275 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:42:41.054191 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e1c383-3ea0-4497-9476-8ed87dbfd8b2 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:41.055208 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e1c383-3ea0-4497-9476-8ed87dbfd8b2 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:41.055677 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e1c383-3ea0-4497-9476-8ed87dbfd8b2 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:41.057243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 28/55] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:41.062983 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfdb823-4085-49f2-b6e7-ac7173ba1d44 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:41.063793 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfdb823-4085-49f2-b6e7-ac7173ba1d44 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:41.064288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfdb823-4085-49f2-b6e7-ac7173ba1d44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:41.084972 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-9dfdb823-4085-49f2-b6e7-ac7173ba1d44 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:41.086456 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-9dfdb823-4085-49f2-b6e7-ac7173ba1d44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:41.142756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:41.168415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 28/56] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:42:41.246187 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f35719-91a7-444a-bca7-eccc92b16989 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:41.246795 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f35719-91a7-444a-bca7-eccc92b16989 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:41.247250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f35719-91a7-444a-bca7-eccc92b16989 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:41.248831 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 29/57] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:41.254412 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968fa4b4-c657-400b-9102-9a252991abce None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:41.254932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968fa4b4-c657-400b-9102-9a252991abce None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:41.255575 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968fa4b4-c657-400b-9102-9a252991abce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:41.257710 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff000c16-a2f3-40a8-8f53-1613e134123b None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:41.258179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff000c16-a2f3-40a8-8f53-1613e134123b None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:41.258626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff000c16-a2f3-40a8-8f53-1613e134123b None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:41.260383 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 30/58] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:41.266551 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc77772-d7e4-4e8c-b2ce-8f4bfcbfaca9 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:41.267088 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc77772-d7e4-4e8c-b2ce-8f4bfcbfaca9 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:41.267578 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc77772-d7e4-4e8c-b2ce-8f4bfcbfaca9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:41.295491 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-5bc77772-d7e4-4e8c-b2ce-8f4bfcbfaca9 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:41.297008 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-5bc77772-d7e4-4e8c-b2ce-8f4bfcbfaca9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:41.300122 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-968fa4b4-c657-400b-9102-9a252991abce None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:41.301526 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-968fa4b4-c657-400b-9102-9a252991abce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:41.347079 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:41.409838 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 29/59] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:42:41.423170 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 31/60] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:42:41.590163 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968fa4b4-c657-400b-9102-9a252991abce None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:41.719421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968fa4b4-c657-400b-9102-9a252991abce None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:41.739337 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc77772-d7e4-4e8c-b2ce-8f4bfcbfaca9 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:41.740211 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:41.782672 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed034021-862a-4fc9-95c9-b174e5c4b905 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:41.783608 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed034021-862a-4fc9-95c9-b174e5c4b905 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:41.784100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed034021-862a-4fc9-95c9-b174e5c4b905 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:41.784513 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed034021-862a-4fc9-95c9-b174e5c4b905 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:41.797067 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed034021-862a-4fc9-95c9-b174e5c4b905 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:42:41.802381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-ed034021-862a-4fc9-95c9-b174e5c4b905 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:42:41.802381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:41.802381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:42:41.802381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:42:41.802381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:42:41.802381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:42:41.802381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:42:41.802381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:42:41.802381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:42:41.802381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:42:41.802381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:42:41.802381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:42:41.802381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:42:41.802381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:42:41.802381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:42:41.802381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:42:41.803299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:42:41.803299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:42:41.803299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:42:41.803299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:42:41.803299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:42:41.803299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:42:41.803299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:42:41.803299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:42:41.803299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:42:41.803299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:42:41.803299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:42:41.803299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:42:41.803299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:42:41.803299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:42:41.803299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:42:41.803299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:42:41.804092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:42:41.804092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:42:41.804092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:42:41.804092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:42:41.804092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:42:41.804092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:42:41.804092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:42:41.804092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:42:41.804092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:42:41.804092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:42:41.804092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:42:41.804092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:42:41.806346 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 30/61] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:42:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:42:41.807929 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed034021-862a-4fc9-95c9-b174e5c4b905 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:41.861787 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc77772-d7e4-4e8c-b2ce-8f4bfcbfaca9 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:41.897144 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90216a14-2d0b-412e-aa45-3633b543622b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:41.897779 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90216a14-2d0b-412e-aa45-3633b543622b None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:41.898056 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90216a14-2d0b-412e-aa45-3633b543622b None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:41.898324 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90216a14-2d0b-412e-aa45-3633b543622b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:41.908160 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90216a14-2d0b-412e-aa45-3633b543622b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:42:41.909779 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed034021-862a-4fc9-95c9-b174e5c4b905 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:41.913121 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-90216a14-2d0b-412e-aa45-3633b543622b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 30 00:42:41.913121 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:41.913121 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:42:41.913121 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:42:41.913121 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:42:41.913121 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:42:41.913121 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:42:41.913121 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:42:41.913121 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:42:41.913121 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:42:41.913121 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:42:41.913121 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:42:41.913121 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:42:41.913121 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:42:41.913121 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:42:41.913121 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:42:41.914027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:42:41.914027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:42:41.914027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:42:41.914027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:42:41.914027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:42:41.914027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:42:41.914027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:42:41.914027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:42:41.914027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:42:41.914027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:42:41.914027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:42:41.914027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:42:41.914027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:42:41.914027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:42:41.914027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:42:41.914027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:42:41.914783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:42:41.914783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:42:41.914783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:42:41.914783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:42:41.914783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:42:41.914783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:42:41.914783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:42:41.914783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:42:41.914783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:42:41.914783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:42:41.914783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 30 00:42:41.914783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:42:41.914783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 32/62] 10.222.0.16 () {56 vars in 1114 bytes} [Sun Oct 30 00:42:41 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:42:41.915972 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9736e77-6d6d-408d-a83e-07c12ee88653 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:41.916255 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9736e77-6d6d-408d-a83e-07c12ee88653 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:41.916528 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9736e77-6d6d-408d-a83e-07c12ee88653 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:41.917767 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 33/63] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:41.918515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9736e77-6d6d-408d-a83e-07c12ee88653 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:41.944350 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:41.950200 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d2c966-1e91-4930-8e7a-8bc20d31b822 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:41.950835 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d2c966-1e91-4930-8e7a-8bc20d31b822 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:41.951130 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d2c966-1e91-4930-8e7a-8bc20d31b822 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:41.951400 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d2c966-1e91-4930-8e7a-8bc20d31b822 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:41.961817 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13d2c966-1e91-4930-8e7a-8bc20d31b822 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:42:41.966632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-13d2c966-1e91-4930-8e7a-8bc20d31b822 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:42:41.966632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:41.966632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:42:41.966632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:42:41.966632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:42:41.966632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:42:41.966632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:42:41.966632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:42:41.966632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:42:41.966632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:42:41.966632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:42:41.966632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:42:41.966632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:42:41.966632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:42:41.966632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:42:41.966632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:42:41.967677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:42:41.967677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:42:41.967677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:42:41.967677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:42:41.967677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:42:41.967677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:42:41.967677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:42:41.967677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:42:41.967677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:42:41.967677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:42:41.967677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:42:41.967677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:42:41.967677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:42:41.967677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:42:41.967677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:42:41.967677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:42:41.968619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:42:41.968619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:42:41.968619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:42:41.968619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:42:41.968619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:42:41.968619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:42:41.968619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:42:41.968619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:42:41.968619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:42:41.968619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:42:41.968619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:42:41.968619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:42:41.968619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 31/64] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:42:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:42:41.970973 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34418206-0c52-4d5c-95fe-c4c01834d9cc None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:41.971491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34418206-0c52-4d5c-95fe-c4c01834d9cc None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:41.971796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34418206-0c52-4d5c-95fe-c4c01834d9cc None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:41.973113 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 32/65] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:41 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:41.976438 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34418206-0c52-4d5c-95fe-c4c01834d9cc None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:42.012697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:42.021476 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9736e77-6d6d-408d-a83e-07c12ee88653 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:42.054525 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d9ebb9-f6a3-4360-af35-d63b5d745676 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:42.055116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d9ebb9-f6a3-4360-af35-d63b5d745676 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:42.055419 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d9ebb9-f6a3-4360-af35-d63b5d745676 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:42.055683 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d9ebb9-f6a3-4360-af35-d63b5d745676 None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:42.056371 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d9ebb9-f6a3-4360-af35-d63b5d745676 None admin] RBAC: Adding query filter params (name=member) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:42.056669 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d9ebb9-f6a3-4360-af35-d63b5d745676 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:42.058058 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d9ebb9-f6a3-4360-af35-d63b5d745676 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:42.066520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 34/66] 10.222.0.16 () {56 vars in 1101 bytes} [Sun Oct 30 00:42:41 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:42.067747 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4749aef-fca9-495d-8ba0-264f9730fcd6 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:42.068041 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4749aef-fca9-495d-8ba0-264f9730fcd6 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:42.068310 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4749aef-fca9-495d-8ba0-264f9730fcd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:42.079444 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34418206-0c52-4d5c-95fe-c4c01834d9cc None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:42.085824 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-f4749aef-fca9-495d-8ba0-264f9730fcd6 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:42.087016 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-f4749aef-fca9-495d-8ba0-264f9730fcd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:42.113745 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2dfa01-7b9d-4f1d-8649-776f4ce5c7d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:42.114346 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2dfa01-7b9d-4f1d-8649-776f4ce5c7d2 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:42.114627 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2dfa01-7b9d-4f1d-8649-776f4ce5c7d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:42.114890 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2dfa01-7b9d-4f1d-8649-776f4ce5c7d2 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:42.115544 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2dfa01-7b9d-4f1d-8649-776f4ce5c7d2 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:42.115877 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2dfa01-7b9d-4f1d-8649-776f4ce5c7d2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:42.117262 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2dfa01-7b9d-4f1d-8649-776f4ce5c7d2 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:42.125215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 33/67] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:42:41 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:42.126090 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2dfa01-7b9d-4f1d-8649-776f4ce5c7d2 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:42.155925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 35/68] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:42:42.157160 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35921e32-0753-41a9-86fb-701497b8c59c None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:42.157433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35921e32-0753-41a9-86fb-701497b8c59c None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:42.157700 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35921e32-0753-41a9-86fb-701497b8c59c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:42.174799 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-35921e32-0753-41a9-86fb-701497b8c59c None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:42.176012 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-35921e32-0753-41a9-86fb-701497b8c59c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:42.187994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:42.239957 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:42.244213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2dfa01-7b9d-4f1d-8649-776f4ce5c7d2 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:42.247587 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 36/69] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:42:42.252590 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35921e32-0753-41a9-86fb-701497b8c59c None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:42.281177 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f3a040-693d-48bc-93bb-0bf96686a923 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:42.282066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f3a040-693d-48bc-93bb-0bf96686a923 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:42.282534 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f3a040-693d-48bc-93bb-0bf96686a923 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:42.282972 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f3a040-693d-48bc-93bb-0bf96686a923 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:42.283858 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f3a040-693d-48bc-93bb-0bf96686a923 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:42.284346 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f3a040-693d-48bc-93bb-0bf96686a923 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:42.286019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f3a040-693d-48bc-93bb-0bf96686a923 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:42.294741 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 34/70] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:42:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:42.300587 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f3a040-693d-48bc-93bb-0bf96686a923 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:42.390798 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35921e32-0753-41a9-86fb-701497b8c59c None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:42.419904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f3a040-693d-48bc-93bb-0bf96686a923 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:42.432819 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8d0283-1d1f-455f-8426-fef47a0dd52b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:42.433450 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8d0283-1d1f-455f-8426-fef47a0dd52b None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:42.433731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8d0283-1d1f-455f-8426-fef47a0dd52b None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:42.433998 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8d0283-1d1f-455f-8426-fef47a0dd52b None admin] PATH_INFO: `/v3/users/1467c783ff2d42a9b97f8ca3d67bea0a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:42.448214 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8d0283-1d1f-455f-8426-fef47a0dd52b None admin] RBAC: Authorizing `identity:get_user(user_id=1467c783ff2d42a9b97f8ca3d67bea0a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:42.449781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8d0283-1d1f-455f-8426-fef47a0dd52b None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:42.451424 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 37/71] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:42:42 2022] GET /identity/v3/users/1467c783ff2d42a9b97f8ca3d67bea0a => generated 275 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:42.454297 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c885f61e-94e5-49ab-8aeb-f62283803470 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:42.454929 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c885f61e-94e5-49ab-8aeb-f62283803470 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:42.455200 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c885f61e-94e5-49ab-8aeb-f62283803470 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:42.455463 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c885f61e-94e5-49ab-8aeb-f62283803470 None admin] PATH_INFO: `/v3/users/1467c783ff2d42a9b97f8ca3d67bea0a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:42.456645 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8d0283-1d1f-455f-8426-fef47a0dd52b None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:42.457119 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c885f61e-94e5-49ab-8aeb-f62283803470 None admin] RBAC: Authorizing `identity:get_user(user_id=1467c783ff2d42a9b97f8ca3d67bea0a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:42.458708 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c885f61e-94e5-49ab-8aeb-f62283803470 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:42.460446 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 35/72] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:42:42 2022] GET /identity/v3/users/1467c783ff2d42a9b97f8ca3d67bea0a => generated 275 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:42.461337 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:42.465688 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c885f61e-94e5-49ab-8aeb-f62283803470 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:42.494964 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:42.556981 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8d0283-1d1f-455f-8426-fef47a0dd52b None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:42.565761 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c885f61e-94e5-49ab-8aeb-f62283803470 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:42.590785 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed9b877-de09-40da-9ed2-82e759b1510a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:42.591410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed9b877-de09-40da-9ed2-82e759b1510a None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:42.591688 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed9b877-de09-40da-9ed2-82e759b1510a None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:42.591985 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed9b877-de09-40da-9ed2-82e759b1510a None admin] PATH_INFO: `/v3/users/ca6c0649e64b4d38bdd14a04f31beb73` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:42.593109 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed9b877-de09-40da-9ed2-82e759b1510a None admin] RBAC: Authorizing `identity:get_user(user_id=ca6c0649e64b4d38bdd14a04f31beb73)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:42.594564 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed9b877-de09-40da-9ed2-82e759b1510a None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:42.596173 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 38/73] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:42:42 2022] GET /identity/v3/users/ca6c0649e64b4d38bdd14a04f31beb73 => generated 247 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:42.596997 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed9b877-de09-40da-9ed2-82e759b1510a None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:42.599233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6825d1d2-ac15-4d60-8b8e-ad5a69e66715 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:42.600008 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6825d1d2-ac15-4d60-8b8e-ad5a69e66715 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:42.600284 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6825d1d2-ac15-4d60-8b8e-ad5a69e66715 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:42.600554 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6825d1d2-ac15-4d60-8b8e-ad5a69e66715 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:42.611230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6825d1d2-ac15-4d60-8b8e-ad5a69e66715 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:42:42.616638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-6825d1d2-ac15-4d60-8b8e-ad5a69e66715 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:42:42.616638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:42.616638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:42:42.616638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:42:42.616638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:42:42.616638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:42:42.616638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:42:42.616638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:42:42.616638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:42:42.616638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:42:42.616638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:42:42.616638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:42:42.616638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:42:42.616638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:42:42.616638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:42:42.616638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:42:42.617520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:42:42.617520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:42:42.617520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:42:42.617520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:42:42.617520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:42:42.617520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:42:42.617520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:42:42.617520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:42:42.617520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:42:42.617520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:42:42.617520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:42:42.617520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:42:42.617520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:42:42.617520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:42:42.617520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:42:42.617520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:42:42.618539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:42:42.618539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:42:42.618539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:42:42.618539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:42:42.618539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:42:42.618539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:42:42.618539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:42:42.618539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:42:42.618539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:42:42.618539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:42:42.618539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:42:42.618539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:42:42.618539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 36/74] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:42:42 2022] GET /identity/v3/roles/member => generated 84 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:42:42.620542 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6825d1d2-ac15-4d60-8b8e-ad5a69e66715 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:42.665643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:42.695652 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed9b877-de09-40da-9ed2-82e759b1510a None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:42.710693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:42.723376 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6825d1d2-ac15-4d60-8b8e-ad5a69e66715 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:42.729930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b714300-844e-4bd2-878f-b0bcc979803d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:42.730558 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b714300-844e-4bd2-878f-b0bcc979803d None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:42.730830 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b714300-844e-4bd2-878f-b0bcc979803d None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:42.731098 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b714300-844e-4bd2-878f-b0bcc979803d None admin] PATH_INFO: `/v3/projects/e48719636cc1460283c3bc75e46b4a17` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:42.732497 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b714300-844e-4bd2-878f-b0bcc979803d None admin] RBAC: Authorizing `identity:get_project(project_id=e48719636cc1460283c3bc75e46b4a17)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:42.733727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b714300-844e-4bd2-878f-b0bcc979803d None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:42.735364 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 39/75] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:42:42 2022] GET /identity/v3/projects/e48719636cc1460283c3bc75e46b4a17 => generated 298 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:42.736138 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b714300-844e-4bd2-878f-b0bcc979803d None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:42.760986 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb7a357-db57-4704-adad-4956f169e530 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:42.761609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb7a357-db57-4704-adad-4956f169e530 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:42.761888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb7a357-db57-4704-adad-4956f169e530 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:42.762188 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb7a357-db57-4704-adad-4956f169e530 None admin] PATH_INFO: `/v3/projects/e48719636cc1460283c3bc75e46b4a17` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:42.763320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb7a357-db57-4704-adad-4956f169e530 None admin] RBAC: Authorizing `identity:get_project(project_id=e48719636cc1460283c3bc75e46b4a17)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:42.764793 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb7a357-db57-4704-adad-4956f169e530 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:42.766432 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 37/76] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:42:42 2022] GET /identity/v3/projects/e48719636cc1460283c3bc75e46b4a17 => generated 298 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:42.767251 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb7a357-db57-4704-adad-4956f169e530 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:42.836585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b714300-844e-4bd2-878f-b0bcc979803d None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:42.866073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb7a357-db57-4704-adad-4956f169e530 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:42.869925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19ece1c-e4e5-49d6-9fe5-af49de2576f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:42.870547 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19ece1c-e4e5-49d6-9fe5-af49de2576f9 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:42.870819 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19ece1c-e4e5-49d6-9fe5-af49de2576f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:42.871084 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19ece1c-e4e5-49d6-9fe5-af49de2576f9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:42.872201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19ece1c-e4e5-49d6-9fe5-af49de2576f9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:42.873618 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19ece1c-e4e5-49d6-9fe5-af49de2576f9 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:42.875759 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 40/77] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:42:42 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:42.875931 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19ece1c-e4e5-49d6-9fe5-af49de2576f9 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:42.887842 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:42.900750 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c5ed39-bf59-42c8-8aeb-12b32d251071 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:42.901395 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c5ed39-bf59-42c8-8aeb-12b32d251071 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:42.901677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c5ed39-bf59-42c8-8aeb-12b32d251071 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:42.901947 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c5ed39-bf59-42c8-8aeb-12b32d251071 None admin] PATH_INFO: `/v3/projects/e48719636cc1460283c3bc75e46b4a17` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:42.903159 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c5ed39-bf59-42c8-8aeb-12b32d251071 None admin] RBAC: Authorizing `identity:get_project(project_id=e48719636cc1460283c3bc75e46b4a17)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:42.904668 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c5ed39-bf59-42c8-8aeb-12b32d251071 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:42.906464 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 38/78] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:42:42 2022] GET /identity/v3/projects/e48719636cc1460283c3bc75e46b4a17 => generated 298 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:42.907033 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c5ed39-bf59-42c8-8aeb-12b32d251071 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:42.932615 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:42.979687 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19ece1c-e4e5-49d6-9fe5-af49de2576f9 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:43.006787 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c5ed39-bf59-42c8-8aeb-12b32d251071 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:43.012835 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a815a5-85cc-4e65-aa73-2058b249668c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:43.013395 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a815a5-85cc-4e65-aa73-2058b249668c None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:43.013669 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a815a5-85cc-4e65-aa73-2058b249668c None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:43.013935 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a815a5-85cc-4e65-aa73-2058b249668c None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:43.014593 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a815a5-85cc-4e65-aa73-2058b249668c None admin] RBAC: Adding query filter params (name=member) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:43.014896 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a815a5-85cc-4e65-aa73-2058b249668c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:43.016318 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a815a5-85cc-4e65-aa73-2058b249668c None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:43.024331 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 41/79] 10.222.0.16 () {56 vars in 1101 bytes} [Sun Oct 30 00:42:42 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:43.025095 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a815a5-85cc-4e65-aa73-2058b249668c None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:43.043341 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453020d5-e605-4452-b27a-99cf822e2c43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:43.044035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453020d5-e605-4452-b27a-99cf822e2c43 None admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:43.044311 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453020d5-e605-4452-b27a-99cf822e2c43 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:43.044592 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453020d5-e605-4452-b27a-99cf822e2c43 None admin] PATH_INFO: `/v3/projects/e48719636cc1460283c3bc75e46b4a17/users/1467c783ff2d42a9b97f8ca3d67bea0a/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:43.046542 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453020d5-e605-4452-b27a-99cf822e2c43 None admin] RBAC: Authorizing `identity:create_grant(project_id=e48719636cc1460283c3bc75e46b4a17, user_id=1467c783ff2d42a9b97f8ca3d67bea0a, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:43.048022 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453020d5-e605-4452-b27a-99cf822e2c43 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:43.062483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 39/80] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:42:42 2022] PUT /identity/v3/projects/e48719636cc1460283c3bc75e46b4a17/users/1467c783ff2d42a9b97f8ca3d67bea0a/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:42:43.063570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453020d5-e605-4452-b27a-99cf822e2c43 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:43.105536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:43.143110 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a815a5-85cc-4e65-aa73-2058b249668c None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:43.156369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:43.175154 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453020d5-e605-4452-b27a-99cf822e2c43 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:43.182509 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5655f125-6351-44bb-9c50-e0cbc6d7de72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:43.183176 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5655f125-6351-44bb-9c50-e0cbc6d7de72 None admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:43.183453 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5655f125-6351-44bb-9c50-e0cbc6d7de72 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:43.183763 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5655f125-6351-44bb-9c50-e0cbc6d7de72 None admin] PATH_INFO: `/v3/projects/e48719636cc1460283c3bc75e46b4a17/users/1467c783ff2d42a9b97f8ca3d67bea0a/roles/22e839a60f6f414abd42c577025184a2` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:43.185986 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5655f125-6351-44bb-9c50-e0cbc6d7de72 None admin] RBAC: Authorizing `identity:create_grant(project_id=e48719636cc1460283c3bc75e46b4a17, user_id=1467c783ff2d42a9b97f8ca3d67bea0a, role_id=22e839a60f6f414abd42c577025184a2)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:43.187779 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5655f125-6351-44bb-9c50-e0cbc6d7de72 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:43.203505 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 42/81] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:42:43 2022] PUT /identity/v3/projects/e48719636cc1460283c3bc75e46b4a17/users/1467c783ff2d42a9b97f8ca3d67bea0a/roles/22e839a60f6f414abd42c577025184a2 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:42:43.205350 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5655f125-6351-44bb-9c50-e0cbc6d7de72 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:43.236127 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b953f225-c2a1-4192-8e42-f0b48b5b101c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:43.237044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b953f225-c2a1-4192-8e42-f0b48b5b101c None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:43.237477 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b953f225-c2a1-4192-8e42-f0b48b5b101c None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:43.237757 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b953f225-c2a1-4192-8e42-f0b48b5b101c None admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:43.238468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b953f225-c2a1-4192-8e42-f0b48b5b101c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:43.241829 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b953f225-c2a1-4192-8e42-f0b48b5b101c None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:43.271132 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 40/82] 10.222.0.16 () {60 vars in 1132 bytes} [Sun Oct 30 00:42:43 2022] POST /identity/v3/users => generated 289 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:42:43.272911 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b953f225-c2a1-4192-8e42-f0b48b5b101c None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:43.351753 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5655f125-6351-44bb-9c50-e0cbc6d7de72 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:43.394147 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:43.409053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09196216-b055-4dc6-8cfc-b1755e015e95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:43.409053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09196216-b055-4dc6-8cfc-b1755e015e95 None admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:43.409053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09196216-b055-4dc6-8cfc-b1755e015e95 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:43.409364 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09196216-b055-4dc6-8cfc-b1755e015e95 None admin] PATH_INFO: `/v3/projects/e48719636cc1460283c3bc75e46b4a17/users/ca6c0649e64b4d38bdd14a04f31beb73/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:43.417344 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09196216-b055-4dc6-8cfc-b1755e015e95 None admin] RBAC: Authorizing `identity:create_grant(project_id=e48719636cc1460283c3bc75e46b4a17, user_id=ca6c0649e64b4d38bdd14a04f31beb73, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:43.420069 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09196216-b055-4dc6-8cfc-b1755e015e95 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:43.434612 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b953f225-c2a1-4192-8e42-f0b48b5b101c None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:43.439963 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 43/83] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:42:43 2022] PUT /identity/v3/projects/e48719636cc1460283c3bc75e46b4a17/users/ca6c0649e64b4d38bdd14a04f31beb73/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:42:43.478567 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2480eddd-f699-4c91-86e3-2879ec815365 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:43.479621 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2480eddd-f699-4c91-86e3-2879ec815365 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:43.480135 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2480eddd-f699-4c91-86e3-2879ec815365 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:43.480596 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2480eddd-f699-4c91-86e3-2879ec815365 None admin] PATH_INFO: `/v3/users/1467c783ff2d42a9b97f8ca3d67bea0a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:43.482184 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2480eddd-f699-4c91-86e3-2879ec815365 None admin] RBAC: Authorizing `identity:get_user(user_id=1467c783ff2d42a9b97f8ca3d67bea0a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:43.484042 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2480eddd-f699-4c91-86e3-2879ec815365 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:43.486159 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 41/84] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:42:43 2022] GET /identity/v3/users/1467c783ff2d42a9b97f8ca3d67bea0a => generated 275 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:43.493090 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09196216-b055-4dc6-8cfc-b1755e015e95 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:43.614812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09196216-b055-4dc6-8cfc-b1755e015e95 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:43.652865 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e018cf-05d3-4e5b-9979-be6c8556aed4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:43.653509 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e018cf-05d3-4e5b-9979-be6c8556aed4 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:43.653788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e018cf-05d3-4e5b-9979-be6c8556aed4 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:43.654059 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e018cf-05d3-4e5b-9979-be6c8556aed4 None admin] PATH_INFO: `/v3/projects/b9a94514dbd54be18ab7776773cf1b65` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:43.655242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e018cf-05d3-4e5b-9979-be6c8556aed4 None admin] RBAC: Authorizing `identity:get_project(project_id=b9a94514dbd54be18ab7776773cf1b65)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:43.656732 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e018cf-05d3-4e5b-9979-be6c8556aed4 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:43.658552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 44/85] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:42:43 2022] GET /identity/v3/projects/b9a94514dbd54be18ab7776773cf1b65 => generated 312 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:43.664542 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2480eddd-f699-4c91-86e3-2879ec815365 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:43.678254 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:43.790311 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2480eddd-f699-4c91-86e3-2879ec815365 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:43.825585 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190f7760-8cd8-4a6d-a398-8fea8bad5acd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:43.826322 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190f7760-8cd8-4a6d-a398-8fea8bad5acd None admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:43.826656 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190f7760-8cd8-4a6d-a398-8fea8bad5acd None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:43.826938 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190f7760-8cd8-4a6d-a398-8fea8bad5acd None admin] PATH_INFO: `/v3/projects/b9a94514dbd54be18ab7776773cf1b65/users/1467c783ff2d42a9b97f8ca3d67bea0a/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:43.828952 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190f7760-8cd8-4a6d-a398-8fea8bad5acd None admin] RBAC: Authorizing `identity:create_grant(project_id=b9a94514dbd54be18ab7776773cf1b65, user_id=1467c783ff2d42a9b97f8ca3d67bea0a, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:43.830610 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190f7760-8cd8-4a6d-a398-8fea8bad5acd None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:43.846117 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 42/86] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:42:43 2022] PUT /identity/v3/projects/b9a94514dbd54be18ab7776773cf1b65/users/1467c783ff2d42a9b97f8ca3d67bea0a/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:42:46.009620 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce40c88-2055-4bfd-b785-8fdb22ba74c7 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:46.010615 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce40c88-2055-4bfd-b785-8fdb22ba74c7 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:46.011084 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce40c88-2055-4bfd-b785-8fdb22ba74c7 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:46.012703 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 45/87] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:46.018284 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fab657-d025-474e-9e8f-c9476dfdcc89 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:46.019144 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fab657-d025-474e-9e8f-c9476dfdcc89 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:46.019782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fab657-d025-474e-9e8f-c9476dfdcc89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:46.045890 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-a6fab657-d025-474e-9e8f-c9476dfdcc89 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:46.047819 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-a6fab657-d025-474e-9e8f-c9476dfdcc89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:46.128226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:46.134761 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 43/88] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:42:46.403053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3cf5b5-4a76-4fb6-841f-5225607cfded None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:46.403671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3cf5b5-4a76-4fb6-841f-5225607cfded None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:46.404157 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3cf5b5-4a76-4fb6-841f-5225607cfded None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:46.405718 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 46/89] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:46.421314 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc7a8e3-edb9-4a20-be98-faebf2246c05 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:46.421886 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc7a8e3-edb9-4a20-be98-faebf2246c05 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:46.422341 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc7a8e3-edb9-4a20-be98-faebf2246c05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:46.449261 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-0fc7a8e3-edb9-4a20-be98-faebf2246c05 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:46.450878 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-0fc7a8e3-edb9-4a20-be98-faebf2246c05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:46.479811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d28c8d-0402-4f4d-9d9b-c23cd1053c35 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:46.480425 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d28c8d-0402-4f4d-9d9b-c23cd1053c35 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:46.480881 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d28c8d-0402-4f4d-9d9b-c23cd1053c35 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:46.482471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 47/90] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:42:46.493606 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b9c22b-7d7d-49e1-858c-3c09e834d5f5 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:46.494209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b9c22b-7d7d-49e1-858c-3c09e834d5f5 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:46.494667 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b9c22b-7d7d-49e1-858c-3c09e834d5f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:46.522651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-e6b9c22b-7d7d-49e1-858c-3c09e834d5f5 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:46.524682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-e6b9c22b-7d7d-49e1-858c-3c09e834d5f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:46.586312 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 44/91] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:42:46.588293 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc7a8e3-edb9-4a20-be98-faebf2246c05 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:46.626958 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:46.643768 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 48/92] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:42:46.645319 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e39c56-8fc3-42a9-a2ef-92dad7aa522a None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:46.645605 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e39c56-8fc3-42a9-a2ef-92dad7aa522a None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:46.645874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e39c56-8fc3-42a9-a2ef-92dad7aa522a None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:46.656187 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 49/93] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:46 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:46.666801 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb311c11-ac92-4a64-a0b5-3197fb1a01ee None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:46.667079 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb311c11-ac92-4a64-a0b5-3197fb1a01ee None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:46.667352 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb311c11-ac92-4a64-a0b5-3197fb1a01ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:46.696802 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-fb311c11-ac92-4a64-a0b5-3197fb1a01ee None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:46.698097 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-fb311c11-ac92-4a64-a0b5-3197fb1a01ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:46.789013 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc7a8e3-edb9-4a20-be98-faebf2246c05 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:46.812743 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 50/94] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:42:46.814009 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c93e96-bfa0-4b61-8ae0-d505f98cc048 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:46.814288 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c93e96-bfa0-4b61-8ae0-d505f98cc048 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:46.814555 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c93e96-bfa0-4b61-8ae0-d505f98cc048 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:46.815805 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 51/95] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:46.824269 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73e073d-7759-4758-82a0-e952fd8cf908 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:46.824548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73e073d-7759-4758-82a0-e952fd8cf908 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:46.824822 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73e073d-7759-4758-82a0-e952fd8cf908 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:46.848372 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:46.865256 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbde8ac-5a7c-4190-b601-fb73be6be650 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:46.865908 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbde8ac-5a7c-4190-b601-fb73be6be650 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:46.866183 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbde8ac-5a7c-4190-b601-fb73be6be650 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:46.866450 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbde8ac-5a7c-4190-b601-fb73be6be650 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:46.868883 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-b73e073d-7759-4758-82a0-e952fd8cf908 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:46.870061 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b73e073d-7759-4758-82a0-e952fd8cf908 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:46.885012 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fbde8ac-5a7c-4190-b601-fb73be6be650 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:42:46.890419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-9fbde8ac-5a7c-4190-b601-fb73be6be650 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 30 00:42:46.890419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:46.890419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:42:46.890419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:42:46.890419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:42:46.890419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:42:46.890419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:42:46.890419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:42:46.890419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:42:46.890419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:42:46.890419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:42:46.890419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:42:46.890419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:42:46.890419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:42:46.890419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:42:46.890419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:42:46.891243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:42:46.891243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:42:46.891243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:42:46.891243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:42:46.891243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:42:46.891243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:42:46.891243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:42:46.891243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:42:46.891243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:42:46.891243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:42:46.891243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:42:46.891243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:42:46.891243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:42:46.891243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:42:46.891243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:42:46.891243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:42:46.892027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:42:46.892027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:42:46.892027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:42:46.892027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:42:46.892027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:42:46.892027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:42:46.892027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:42:46.892027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:42:46.892027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:42:46.892027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:42:46.892027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 30 00:42:46.892027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:42:46.894294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 45/96] 10.222.0.16 () {56 vars in 1114 bytes} [Sun Oct 30 00:42:46 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:42:46.900152 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbde8ac-5a7c-4190-b601-fb73be6be650 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:46.941292 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 52/97] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:42:47.006353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbde8ac-5a7c-4190-b601-fb73be6be650 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:47.023562 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73e073d-7759-4758-82a0-e952fd8cf908 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:47.041654 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad663d2-3914-4cb0-99a8-0499b3af5aa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:47.043149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad663d2-3914-4cb0-99a8-0499b3af5aa4 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:47.043640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad663d2-3914-4cb0-99a8-0499b3af5aa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:47.044124 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad663d2-3914-4cb0-99a8-0499b3af5aa4 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:47.045015 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad663d2-3914-4cb0-99a8-0499b3af5aa4 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:47.045519 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad663d2-3914-4cb0-99a8-0499b3af5aa4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:47.047064 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad663d2-3914-4cb0-99a8-0499b3af5aa4 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:47.056916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 46/98] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:42:46 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:47.064723 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad663d2-3914-4cb0-99a8-0499b3af5aa4 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:47.118321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:47.128442 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:47.183556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad663d2-3914-4cb0-99a8-0499b3af5aa4 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:47.204138 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73e073d-7759-4758-82a0-e952fd8cf908 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:47.223995 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c04320-c161-4a41-94a7-8d10fd52501e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:47.225039 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c04320-c161-4a41-94a7-8d10fd52501e None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:47.225634 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c04320-c161-4a41-94a7-8d10fd52501e None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:47.226090 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c04320-c161-4a41-94a7-8d10fd52501e None admin] PATH_INFO: `/v3/users/1467c783ff2d42a9b97f8ca3d67bea0a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:47.228164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c04320-c161-4a41-94a7-8d10fd52501e None admin] RBAC: Authorizing `identity:get_user(user_id=1467c783ff2d42a9b97f8ca3d67bea0a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:47.229839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c04320-c161-4a41-94a7-8d10fd52501e None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:47.231748 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 47/99] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:42:47 2022] GET /identity/v3/users/1467c783ff2d42a9b97f8ca3d67bea0a => generated 275 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:47.233231 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c04320-c161-4a41-94a7-8d10fd52501e None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:47.240864 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a6d033-d4e5-488d-88ce-cb64882f3253 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:47.241504 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a6d033-d4e5-488d-88ce-cb64882f3253 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:47.241779 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a6d033-d4e5-488d-88ce-cb64882f3253 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:47.242047 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a6d033-d4e5-488d-88ce-cb64882f3253 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:47.252722 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a6d033-d4e5-488d-88ce-cb64882f3253 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:42:47.257925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-d9a6d033-d4e5-488d-88ce-cb64882f3253 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:42:47.257925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:47.257925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:42:47.257925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:42:47.257925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:42:47.257925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:42:47.257925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:42:47.257925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:42:47.257925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:42:47.257925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:42:47.257925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:42:47.257925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:42:47.257925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:42:47.257925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:42:47.257925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:42:47.257925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:42:47.258769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:42:47.258769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:42:47.258769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:42:47.258769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:42:47.258769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:42:47.258769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:42:47.258769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:42:47.258769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:42:47.258769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:42:47.258769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:42:47.258769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:42:47.258769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:42:47.258769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:42:47.258769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:42:47.258769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:42:47.258769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:42:47.259514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:42:47.259514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:42:47.259514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:42:47.259514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:42:47.259514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:42:47.259514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:42:47.259514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:42:47.259514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:42:47.259514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:42:47.259514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:42:47.259514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:42:47.259514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:42:47.259514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 53/100] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:42:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:42:47.260145 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a6d033-d4e5-488d-88ce-cb64882f3253 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:47.348534 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c04320-c161-4a41-94a7-8d10fd52501e None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:47.365330 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a6d033-d4e5-488d-88ce-cb64882f3253 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:47.370919 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:47.376234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:47.392287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202e96cd-6ee6-4a7f-b0ee-c4fafc778826 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:47.392925 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202e96cd-6ee6-4a7f-b0ee-c4fafc778826 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:47.393350 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202e96cd-6ee6-4a7f-b0ee-c4fafc778826 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:47.393627 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202e96cd-6ee6-4a7f-b0ee-c4fafc778826 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:47.401075 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de806f70-2243-4107-817d-e63547457756 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:47.401702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de806f70-2243-4107-817d-e63547457756 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:47.401977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de806f70-2243-4107-817d-e63547457756 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:47.402435 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de806f70-2243-4107-817d-e63547457756 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:47.405452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202e96cd-6ee6-4a7f-b0ee-c4fafc778826 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:42:47.412424 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de806f70-2243-4107-817d-e63547457756 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:42:47.412553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-202e96cd-6ee6-4a7f-b0ee-c4fafc778826 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:42:47.412553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:47.412553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:42:47.412553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:42:47.412553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:42:47.412553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:42:47.412553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:42:47.412553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:42:47.412553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:42:47.412553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:42:47.412553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:42:47.412553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:42:47.412553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:42:47.412553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:42:47.412553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:42:47.412553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:42:47.413355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:42:47.413355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:42:47.413355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:42:47.413355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:42:47.413355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:42:47.413355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:42:47.413355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:42:47.413355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:42:47.413355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:42:47.413355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:42:47.413355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:42:47.413355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:42:47.413355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:42:47.413355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:42:47.413355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:42:47.413355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:42:47.414555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:42:47.414555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:42:47.414555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:42:47.414555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:42:47.414555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:42:47.414555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:42:47.414555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:42:47.414555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:42:47.414555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:42:47.414555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:42:47.414555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:42:47.414555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:42:47.414555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 48/101] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:42:47 2022] GET /identity/v3/roles/member => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:42:47.416578 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202e96cd-6ee6-4a7f-b0ee-c4fafc778826 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:47.417137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-de806f70-2243-4107-817d-e63547457756 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 30 00:42:47.417137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:47.417137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:42:47.417137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:42:47.417137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:42:47.417137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:42:47.417137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:42:47.417137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:42:47.417137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:42:47.417137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:42:47.417137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:42:47.417137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:42:47.417137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:42:47.417137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:42:47.417137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:42:47.417137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:42:47.417969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:42:47.417969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:42:47.417969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:42:47.417969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:42:47.417969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:42:47.417969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:42:47.417969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:42:47.417969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:42:47.417969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:42:47.417969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:42:47.417969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:42:47.417969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:42:47.417969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:42:47.417969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:42:47.417969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:42:47.417969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:42:47.423891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:42:47.423891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:42:47.423891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:42:47.423891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:42:47.423891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:42:47.423891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:42:47.423891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:42:47.423891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:42:47.423891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:42:47.423891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:42:47.423891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 30 00:42:47.423891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:42:47.423891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 54/102] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:42:47 2022] GET /identity/v3/roles/reader => generated 84 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:42:47.423891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de806f70-2243-4107-817d-e63547457756 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:47.525320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202e96cd-6ee6-4a7f-b0ee-c4fafc778826 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:47.531596 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de806f70-2243-4107-817d-e63547457756 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:47.560495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d433cf36-961f-4038-86a2-8abe2d9e838e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:47.561114 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d433cf36-961f-4038-86a2-8abe2d9e838e None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:47.561714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d433cf36-961f-4038-86a2-8abe2d9e838e None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:47.561995 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d433cf36-961f-4038-86a2-8abe2d9e838e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:47.566849 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93e1a2a-2c35-4d60-8d99-fa557896d4d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:47.567668 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93e1a2a-2c35-4d60-8d99-fa557896d4d2 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:47.568095 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93e1a2a-2c35-4d60-8d99-fa557896d4d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:47.568495 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93e1a2a-2c35-4d60-8d99-fa557896d4d2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:47.579032 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93e1a2a-2c35-4d60-8d99-fa557896d4d2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:42:47.584990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-e93e1a2a-2c35-4d60-8d99-fa557896d4d2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:42:47.584990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:47.584990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:42:47.584990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:42:47.584990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:42:47.584990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:42:47.584990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:42:47.584990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:42:47.584990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:42:47.584990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:42:47.584990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:42:47.584990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:42:47.584990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:42:47.584990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:42:47.584990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:42:47.584990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:42:47.586104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:42:47.586104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:42:47.586104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:42:47.586104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:42:47.586104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:42:47.586104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:42:47.586104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:42:47.586104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:42:47.586104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:42:47.586104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:42:47.586104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:42:47.586104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:42:47.586104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:42:47.586104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:42:47.586104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:42:47.586104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:42:47.587038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:42:47.587038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:42:47.587038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:42:47.587038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:42:47.587038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:42:47.587038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:42:47.587038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:42:47.587038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:42:47.587038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:42:47.587038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:42:47.587038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:42:47.587038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:42:47.588942 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 55/103] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:42:47 2022] GET /identity/v3/roles/member => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:42:47.589848 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93e1a2a-2c35-4d60-8d99-fa557896d4d2 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:47.612355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:47.620048 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:47.636300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d433cf36-961f-4038-86a2-8abe2d9e838e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:47.637794 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d433cf36-961f-4038-86a2-8abe2d9e838e None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:47.692629 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93e1a2a-2c35-4d60-8d99-fa557896d4d2 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:47.725356 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e776a319-c9c5-449e-ab89-5986dbcd62e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:47.725963 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e776a319-c9c5-449e-ab89-5986dbcd62e2 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:47.726239 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e776a319-c9c5-449e-ab89-5986dbcd62e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:47.726513 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e776a319-c9c5-449e-ab89-5986dbcd62e2 None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:47.727163 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e776a319-c9c5-449e-ab89-5986dbcd62e2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:47.727472 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e776a319-c9c5-449e-ab89-5986dbcd62e2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:47.728899 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e776a319-c9c5-449e-ab89-5986dbcd62e2 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:47.731652 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 49/104] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:42:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 30 00:42:47.732503 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d433cf36-961f-4038-86a2-8abe2d9e838e None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:47.738671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 56/105] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:42:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:47.739512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e776a319-c9c5-449e-ab89-5986dbcd62e2 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:47.809404 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:47.830809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:47.831122 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d433cf36-961f-4038-86a2-8abe2d9e838e None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:47.841941 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e776a319-c9c5-449e-ab89-5986dbcd62e2 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:47.865715 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d610ae7-8ea5-4994-bf03-483eeda602b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:47.866328 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d610ae7-8ea5-4994-bf03-483eeda602b1 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:47.866607 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d610ae7-8ea5-4994-bf03-483eeda602b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:47.866881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d610ae7-8ea5-4994-bf03-483eeda602b1 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:47.867539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d610ae7-8ea5-4994-bf03-483eeda602b1 None admin] RBAC: Adding query filter params (name=member) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:47.867870 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d610ae7-8ea5-4994-bf03-483eeda602b1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:47.869250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d610ae7-8ea5-4994-bf03-483eeda602b1 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:47.876448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29b8fd3-1014-4141-9f57-fe75d5081a22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:47.877085 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29b8fd3-1014-4141-9f57-fe75d5081a22 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:47.877199 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 50/106] 10.222.0.16 () {56 vars in 1101 bytes} [Sun Oct 30 00:42:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:47.877370 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29b8fd3-1014-4141-9f57-fe75d5081a22 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:47.877646 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29b8fd3-1014-4141-9f57-fe75d5081a22 None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:47.877869 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d610ae7-8ea5-4994-bf03-483eeda602b1 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:47.878311 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29b8fd3-1014-4141-9f57-fe75d5081a22 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:47.878663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29b8fd3-1014-4141-9f57-fe75d5081a22 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:47.880450 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29b8fd3-1014-4141-9f57-fe75d5081a22 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:47.893355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 57/107] 10.222.0.16 () {56 vars in 1101 bytes} [Sun Oct 30 00:42:47 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:47.893355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d83454d-3e08-408f-a84a-f9ae81fa370f None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:47.893355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d83454d-3e08-408f-a84a-f9ae81fa370f None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:47.893355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d83454d-3e08-408f-a84a-f9ae81fa370f None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:47.893355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 58/108] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:47 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:47.893355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d83454d-3e08-408f-a84a-f9ae81fa370f None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:47.980367 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d610ae7-8ea5-4994-bf03-483eeda602b1 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:47.993496 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d83454d-3e08-408f-a84a-f9ae81fa370f None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:48.014365 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4381ec8-bc6d-4b10-9860-57a5315875b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:48.014971 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4381ec8-bc6d-4b10-9860-57a5315875b9 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:48.015247 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4381ec8-bc6d-4b10-9860-57a5315875b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:48.015521 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4381ec8-bc6d-4b10-9860-57a5315875b9 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:48.016202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4381ec8-bc6d-4b10-9860-57a5315875b9 None admin] RBAC: Adding query filter params (name=member) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:48.016500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4381ec8-bc6d-4b10-9860-57a5315875b9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:48.017879 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4381ec8-bc6d-4b10-9860-57a5315875b9 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:48.025946 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 51/109] 10.222.0.16 () {56 vars in 1101 bytes} [Sun Oct 30 00:42:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:48.025946 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4381ec8-bc6d-4b10-9860-57a5315875b9 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:48.031092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e79bb0-84fc-4b0e-a26f-09258c601165 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:48.031574 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:48.031706 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e79bb0-84fc-4b0e-a26f-09258c601165 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:48.031706 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e79bb0-84fc-4b0e-a26f-09258c601165 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:48.031706 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e79bb0-84fc-4b0e-a26f-09258c601165 None admin] PATH_INFO: `/v3/projects/e48719636cc1460283c3bc75e46b4a17` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:48.031706 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e79bb0-84fc-4b0e-a26f-09258c601165 None admin] RBAC: Authorizing `identity:get_project(project_id=e48719636cc1460283c3bc75e46b4a17)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:48.033130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e79bb0-84fc-4b0e-a26f-09258c601165 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:48.034770 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 59/110] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:42:47 2022] GET /identity/v3/projects/e48719636cc1460283c3bc75e46b4a17 => generated 298 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:48.035635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e79bb0-84fc-4b0e-a26f-09258c601165 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:48.060079 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:48.127016 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4381ec8-bc6d-4b10-9860-57a5315875b9 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:48.137643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e79bb0-84fc-4b0e-a26f-09258c601165 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:48.160862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8598ac62-afd4-4268-9bf1-63c161f1da22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:48.161493 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8598ac62-afd4-4268-9bf1-63c161f1da22 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:48.161765 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8598ac62-afd4-4268-9bf1-63c161f1da22 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:48.162029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8598ac62-afd4-4268-9bf1-63c161f1da22 None admin] PATH_INFO: `/v3/users/ca6c0649e64b4d38bdd14a04f31beb73` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:48.163172 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8598ac62-afd4-4268-9bf1-63c161f1da22 None admin] RBAC: Authorizing `identity:get_user(user_id=ca6c0649e64b4d38bdd14a04f31beb73)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:48.164650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8598ac62-afd4-4268-9bf1-63c161f1da22 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:48.166205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 52/111] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:42:48 2022] GET /identity/v3/users/ca6c0649e64b4d38bdd14a04f31beb73 => generated 247 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:48.166948 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8598ac62-afd4-4268-9bf1-63c161f1da22 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:48.170842 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b98268-82e0-4688-ad82-7176b8af094c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:48.171510 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b98268-82e0-4688-ad82-7176b8af094c None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:48.171812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b98268-82e0-4688-ad82-7176b8af094c None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:48.172086 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b98268-82e0-4688-ad82-7176b8af094c None admin] PATH_INFO: `/v3/users/1467c783ff2d42a9b97f8ca3d67bea0a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:48.173213 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b98268-82e0-4688-ad82-7176b8af094c None admin] RBAC: Authorizing `identity:get_user(user_id=1467c783ff2d42a9b97f8ca3d67bea0a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:48.174655 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b98268-82e0-4688-ad82-7176b8af094c None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:48.176422 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 60/112] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:42:48 2022] GET /identity/v3/users/1467c783ff2d42a9b97f8ca3d67bea0a => generated 275 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:48.181052 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6722d6f-3be5-4fdb-a85b-6b3dfcf9e6eb None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:48.181052 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6722d6f-3be5-4fdb-a85b-6b3dfcf9e6eb None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:48.181052 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6722d6f-3be5-4fdb-a85b-6b3dfcf9e6eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:48.196296 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-f6722d6f-3be5-4fdb-a85b-6b3dfcf9e6eb None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:48.197478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-f6722d6f-3be5-4fdb-a85b-6b3dfcf9e6eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:48.234166 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:48.264668 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8598ac62-afd4-4268-9bf1-63c161f1da22 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:48.265566 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 61/113] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:42:48.266392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6722d6f-3be5-4fdb-a85b-6b3dfcf9e6eb None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:48.294230 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:48.301340 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e7d1a7-0556-4d2e-9123-0c447ddaa78e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:48.301974 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e7d1a7-0556-4d2e-9123-0c447ddaa78e None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:48.302248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e7d1a7-0556-4d2e-9123-0c447ddaa78e None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:48.302574 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e7d1a7-0556-4d2e-9123-0c447ddaa78e None admin] PATH_INFO: `/v3/users/9afe9a402b7641458693ec64fcd77589` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:48.317896 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e7d1a7-0556-4d2e-9123-0c447ddaa78e None admin] RBAC: Authorizing `identity:get_user(user_id=9afe9a402b7641458693ec64fcd77589)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:48.319254 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e7d1a7-0556-4d2e-9123-0c447ddaa78e None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:48.321074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 53/114] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:42:48 2022] GET /identity/v3/users/9afe9a402b7641458693ec64fcd77589 => generated 289 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:48.321652 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e7d1a7-0556-4d2e-9123-0c447ddaa78e None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:48.388592 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6722d6f-3be5-4fdb-a85b-6b3dfcf9e6eb None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:48.424601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47acc6f9-e0f8-49c4-b202-a7cfd919d5c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:48.425484 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47acc6f9-e0f8-49c4-b202-a7cfd919d5c8 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:48.425874 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e7d1a7-0556-4d2e-9123-0c447ddaa78e None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:48.426151 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47acc6f9-e0f8-49c4-b202-a7cfd919d5c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:48.426560 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47acc6f9-e0f8-49c4-b202-a7cfd919d5c8 None admin] PATH_INFO: `/v3/users/1467c783ff2d42a9b97f8ca3d67bea0a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:48.427870 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47acc6f9-e0f8-49c4-b202-a7cfd919d5c8 None admin] RBAC: Authorizing `identity:get_user(user_id=1467c783ff2d42a9b97f8ca3d67bea0a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:48.429490 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47acc6f9-e0f8-49c4-b202-a7cfd919d5c8 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:48.431218 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 62/115] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:42:48 2022] GET /identity/v3/users/1467c783ff2d42a9b97f8ca3d67bea0a => generated 275 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:48.432149 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47acc6f9-e0f8-49c4-b202-a7cfd919d5c8 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:48.456875 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:48.465525 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fc14f9-c343-4da5-b06f-b7010d0389dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:48.466135 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fc14f9-c343-4da5-b06f-b7010d0389dc None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:48.466460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fc14f9-c343-4da5-b06f-b7010d0389dc None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:48.466758 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fc14f9-c343-4da5-b06f-b7010d0389dc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:48.467473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35fc14f9-c343-4da5-b06f-b7010d0389dc None admin] RBAC: Adding query filter params (role.id=22e839a60f6f414abd42c577025184a2, scope.project.id=e48719636cc1460283c3bc75e46b4a17, user.id=1467c783ff2d42a9b97f8ca3d67bea0a) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:48.467830 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35fc14f9-c343-4da5-b06f-b7010d0389dc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:48.469262 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35fc14f9-c343-4da5-b06f-b7010d0389dc None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:48.475139 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 54/116] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:42:48 2022] GET /identity/v3/role_assignments?user.id=1467c783ff2d42a9b97f8ca3d67bea0a&scope.project.id=e48719636cc1460283c3bc75e46b4a17&role.id=22e839a60f6f414abd42c577025184a2 => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:48.475732 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fc14f9-c343-4da5-b06f-b7010d0389dc None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:48.537942 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:48.556172 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47acc6f9-e0f8-49c4-b202-a7cfd919d5c8 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:48.597288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fc14f9-c343-4da5-b06f-b7010d0389dc None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:48.597819 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7934c5-c5ec-4c44-9497-742cf29f4247 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:48.598451 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7934c5-c5ec-4c44-9497-742cf29f4247 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:48.598804 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7934c5-c5ec-4c44-9497-742cf29f4247 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:48.598987 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7934c5-c5ec-4c44-9497-742cf29f4247 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:48.640391 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a44f77d-b81b-42d3-825e-3f93f20c6bb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:48.641012 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a44f77d-b81b-42d3-825e-3f93f20c6bb0 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:48.641290 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a44f77d-b81b-42d3-825e-3f93f20c6bb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:48.641562 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a44f77d-b81b-42d3-825e-3f93f20c6bb0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:48.685148 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7934c5-c5ec-4c44-9497-742cf29f4247 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:48.686929 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7934c5-c5ec-4c44-9497-742cf29f4247 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:48.712461 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:48.724020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a44f77d-b81b-42d3-825e-3f93f20c6bb0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:48.725500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a44f77d-b81b-42d3-825e-3f93f20c6bb0 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:48.779305 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:48.795615 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 63/117] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:42:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 30 00:42:48.800388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7934c5-c5ec-4c44-9497-742cf29f4247 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:48.822678 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 55/118] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:42:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 30 00:42:48.827261 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a44f77d-b81b-42d3-825e-3f93f20c6bb0 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:48.894761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7934c5-c5ec-4c44-9497-742cf29f4247 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:48.911373 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:48.920963 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a44f77d-b81b-42d3-825e-3f93f20c6bb0 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:48.928285 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7c57c5-a628-4838-8377-5349fe6c6a63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:48.928915 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7c57c5-a628-4838-8377-5349fe6c6a63 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:48.929188 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7c57c5-a628-4838-8377-5349fe6c6a63 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:48.929454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7c57c5-a628-4838-8377-5349fe6c6a63 None admin] PATH_INFO: `/v3/projects/e48719636cc1460283c3bc75e46b4a17` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:48.930584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7c57c5-a628-4838-8377-5349fe6c6a63 None admin] RBAC: Authorizing `identity:get_project(project_id=e48719636cc1460283c3bc75e46b4a17)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:48.932055 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7c57c5-a628-4838-8377-5349fe6c6a63 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:48.933690 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 64/119] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:42:48 2022] GET /identity/v3/projects/e48719636cc1460283c3bc75e46b4a17 => generated 298 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:48.934435 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7c57c5-a628-4838-8377-5349fe6c6a63 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:48.961256 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce65c9f9-e6b8-4c4f-978b-97b31e2ba8bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:48.961859 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce65c9f9-e6b8-4c4f-978b-97b31e2ba8bb None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:48.962135 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce65c9f9-e6b8-4c4f-978b-97b31e2ba8bb None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:48.962398 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce65c9f9-e6b8-4c4f-978b-97b31e2ba8bb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:48.989850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:49.029252 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce65c9f9-e6b8-4c4f-978b-97b31e2ba8bb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:49.030779 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce65c9f9-e6b8-4c4f-978b-97b31e2ba8bb None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:49.032536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7c57c5-a628-4838-8377-5349fe6c6a63 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:49.066059 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290abba1-ff52-43be-91e8-f8ff1c5a8d98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:49.066703 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290abba1-ff52-43be-91e8-f8ff1c5a8d98 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:49.066978 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290abba1-ff52-43be-91e8-f8ff1c5a8d98 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:49.067246 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290abba1-ff52-43be-91e8-f8ff1c5a8d98 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:49.068362 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290abba1-ff52-43be-91e8-f8ff1c5a8d98 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:49.069797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290abba1-ff52-43be-91e8-f8ff1c5a8d98 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:49.071376 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 65/120] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:42:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:49.072168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290abba1-ff52-43be-91e8-f8ff1c5a8d98 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:49.121130 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:49.130224 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 56/121] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:42:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 30 00:42:49.135427 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce65c9f9-e6b8-4c4f-978b-97b31e2ba8bb None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:49.181173 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290abba1-ff52-43be-91e8-f8ff1c5a8d98 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:49.203989 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:49.216340 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635bc90a-f853-406f-a74f-46efff03e717 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:49.216965 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635bc90a-f853-406f-a74f-46efff03e717 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:49.217228 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635bc90a-f853-406f-a74f-46efff03e717 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:49.217515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635bc90a-f853-406f-a74f-46efff03e717 None admin] PATH_INFO: `/v3/projects/e48719636cc1460283c3bc75e46b4a17` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:49.218622 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635bc90a-f853-406f-a74f-46efff03e717 None admin] RBAC: Authorizing `identity:get_project(project_id=e48719636cc1460283c3bc75e46b4a17)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:49.220097 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635bc90a-f853-406f-a74f-46efff03e717 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:49.221739 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 66/122] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:42:49 2022] GET /identity/v3/projects/e48719636cc1460283c3bc75e46b4a17 => generated 298 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:49.222542 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635bc90a-f853-406f-a74f-46efff03e717 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:49.230392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce65c9f9-e6b8-4c4f-978b-97b31e2ba8bb None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:49.264858 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141ce7db-4607-4a8e-a6f9-566bf04f53d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:49.265482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141ce7db-4607-4a8e-a6f9-566bf04f53d9 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:49.265758 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141ce7db-4607-4a8e-a6f9-566bf04f53d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:49.266059 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141ce7db-4607-4a8e-a6f9-566bf04f53d9 None admin] PATH_INFO: `/v3/projects/e48719636cc1460283c3bc75e46b4a17` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:49.267182 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-141ce7db-4607-4a8e-a6f9-566bf04f53d9 None admin] RBAC: Authorizing `identity:get_project(project_id=e48719636cc1460283c3bc75e46b4a17)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:49.268658 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-141ce7db-4607-4a8e-a6f9-566bf04f53d9 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:49.270319 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 57/123] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:42:49 2022] GET /identity/v3/projects/e48719636cc1460283c3bc75e46b4a17 => generated 298 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:49.271052 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141ce7db-4607-4a8e-a6f9-566bf04f53d9 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:49.327659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635bc90a-f853-406f-a74f-46efff03e717 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:49.333845 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:49.361939 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba061c6d-ba6f-461a-a07a-2e61ebc8b9c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:49.362596 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba061c6d-ba6f-461a-a07a-2e61ebc8b9c6 None admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:49.362870 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba061c6d-ba6f-461a-a07a-2e61ebc8b9c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:49.363146 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba061c6d-ba6f-461a-a07a-2e61ebc8b9c6 None admin] PATH_INFO: `/v3/projects/e48719636cc1460283c3bc75e46b4a17/users/9afe9a402b7641458693ec64fcd77589/roles/d330780cbb4e422991f4d18c0970c0b2` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:49.365064 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba061c6d-ba6f-461a-a07a-2e61ebc8b9c6 None admin] RBAC: Authorizing `identity:create_grant(project_id=e48719636cc1460283c3bc75e46b4a17, user_id=9afe9a402b7641458693ec64fcd77589, role_id=d330780cbb4e422991f4d18c0970c0b2)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:49.366491 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba061c6d-ba6f-461a-a07a-2e61ebc8b9c6 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:49.377061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141ce7db-4607-4a8e-a6f9-566bf04f53d9 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:49.379205 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 67/124] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:42:49 2022] PUT /identity/v3/projects/e48719636cc1460283c3bc75e46b4a17/users/9afe9a402b7641458693ec64fcd77589/roles/d330780cbb4e422991f4d18c0970c0b2 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:42:49.380032 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba061c6d-ba6f-461a-a07a-2e61ebc8b9c6 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:49.425243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37793285-3f5a-4736-a233-9713bace79b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:49.426017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37793285-3f5a-4736-a233-9713bace79b2 None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:49.426355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37793285-3f5a-4736-a233-9713bace79b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:49.426696 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37793285-3f5a-4736-a233-9713bace79b2 None admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:49.427415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37793285-3f5a-4736-a233-9713bace79b2 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:49.428984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37793285-3f5a-4736-a233-9713bace79b2 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:49.435802 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:49.471749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 58/125] 10.222.0.16 () {60 vars in 1141 bytes} [Sun Oct 30 00:42:49 2022] POST /identity/v3/projects => generated 302 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:42:49.471749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37793285-3f5a-4736-a233-9713bace79b2 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:49.516747 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba061c6d-ba6f-461a-a07a-2e61ebc8b9c6 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:49.563056 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085bcb28-aacc-4da4-9137-cce4afa5a918 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:49.564759 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085bcb28-aacc-4da4-9137-cce4afa5a918 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:49.565196 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085bcb28-aacc-4da4-9137-cce4afa5a918 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:49.565403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085bcb28-aacc-4da4-9137-cce4afa5a918 None admin] PATH_INFO: `/v3/projects/b9a94514dbd54be18ab7776773cf1b65` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:49.567091 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085bcb28-aacc-4da4-9137-cce4afa5a918 None admin] RBAC: Authorizing `identity:get_project(project_id=b9a94514dbd54be18ab7776773cf1b65)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:49.568669 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085bcb28-aacc-4da4-9137-cce4afa5a918 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:49.570632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 68/126] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:42:49 2022] GET /identity/v3/projects/b9a94514dbd54be18ab7776773cf1b65 => generated 312 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:49.571224 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085bcb28-aacc-4da4-9137-cce4afa5a918 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:49.599665 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37793285-3f5a-4736-a233-9713bace79b2 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:49.630551 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:49.642912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00e9b1f-c1d4-4826-9432-149693204473 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:49.643982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00e9b1f-c1d4-4826-9432-149693204473 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:49.644356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00e9b1f-c1d4-4826-9432-149693204473 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:49.644826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00e9b1f-c1d4-4826-9432-149693204473 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:49.646057 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00e9b1f-c1d4-4826-9432-149693204473 None admin] RBAC: Adding query filter params (role.id=35f59686b5d84291a8d9baf19287eb5a, scope.project.id=e48719636cc1460283c3bc75e46b4a17, user.id=ca6c0649e64b4d38bdd14a04f31beb73) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:49.646982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00e9b1f-c1d4-4826-9432-149693204473 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:49.648961 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00e9b1f-c1d4-4826-9432-149693204473 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:49.658412 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 59/127] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:42:49 2022] GET /identity/v3/role_assignments?user.id=ca6c0649e64b4d38bdd14a04f31beb73&scope.project.id=e48719636cc1460283c3bc75e46b4a17&role.id=35f59686b5d84291a8d9baf19287eb5a => generated 608 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:49.660419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00e9b1f-c1d4-4826-9432-149693204473 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:49.697642 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085bcb28-aacc-4da4-9137-cce4afa5a918 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:49.719446 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:49.750107 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e97cca5-e2ea-406d-a741-d3f16c53dc81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:49.750729 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e97cca5-e2ea-406d-a741-d3f16c53dc81 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:49.751003 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e97cca5-e2ea-406d-a741-d3f16c53dc81 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:49.751270 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e97cca5-e2ea-406d-a741-d3f16c53dc81 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:49.752703 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e97cca5-e2ea-406d-a741-d3f16c53dc81 None admin] RBAC: Adding query filter params (role.id=5c19ef44708c4d1f95f5f657c5c79361, scope.project.id=e48719636cc1460283c3bc75e46b4a17, user.id=1467c783ff2d42a9b97f8ca3d67bea0a) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:49.752703 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e97cca5-e2ea-406d-a741-d3f16c53dc81 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:49.754068 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e97cca5-e2ea-406d-a741-d3f16c53dc81 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:49.769067 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 69/128] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:42:49 2022] GET /identity/v3/role_assignments?user.id=1467c783ff2d42a9b97f8ca3d67bea0a&scope.project.id=e48719636cc1460283c3bc75e46b4a17&role.id=5c19ef44708c4d1f95f5f657c5c79361 => generated 608 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:49.803876 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00e9b1f-c1d4-4826-9432-149693204473 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:49.845153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fc4749-5d14-479e-a500-2a4924411334 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:49.845780 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fc4749-5d14-479e-a500-2a4924411334 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:49.846109 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fc4749-5d14-479e-a500-2a4924411334 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:49.846380 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fc4749-5d14-479e-a500-2a4924411334 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:49.847129 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fc4749-5d14-479e-a500-2a4924411334 None admin] RBAC: Adding query filter params (role.id=5c19ef44708c4d1f95f5f657c5c79361, scope.project.id=b9a94514dbd54be18ab7776773cf1b65, user.id=1467c783ff2d42a9b97f8ca3d67bea0a) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:49.847431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fc4749-5d14-479e-a500-2a4924411334 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:49.849056 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fc4749-5d14-479e-a500-2a4924411334 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:49.856547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 60/129] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:42:49 2022] GET /identity/v3/role_assignments?user.id=1467c783ff2d42a9b97f8ca3d67bea0a&scope.project.id=b9a94514dbd54be18ab7776773cf1b65&role.id=5c19ef44708c4d1f95f5f657c5c79361 => generated 608 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:51.607739 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45be494c-eece-4d8b-a430-bf8172875133 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:51.608293 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45be494c-eece-4d8b-a430-bf8172875133 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:51.608293 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45be494c-eece-4d8b-a430-bf8172875133 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:51.609530 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 70/130] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:51.614851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bfdeab-c47c-4bf0-b4a3-bf979499d4f4 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:51.615299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bfdeab-c47c-4bf0-b4a3-bf979499d4f4 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:51.615387 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bfdeab-c47c-4bf0-b4a3-bf979499d4f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:51.633828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-80bfdeab-c47c-4bf0-b4a3-bf979499d4f4 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:51.634987 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-80bfdeab-c47c-4bf0-b4a3-bf979499d4f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:51.669468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:51.703680 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 61/131] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:42:51.833259 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932fbcc6-22d7-4531-9b03-a12785425ab6 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:51.833566 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932fbcc6-22d7-4531-9b03-a12785425ab6 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:51.833805 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932fbcc6-22d7-4531-9b03-a12785425ab6 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:51.835172 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 71/132] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:51.840672 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507628a9-9a31-4521-8428-5683ae0d04f5 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:51.840958 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507628a9-9a31-4521-8428-5683ae0d04f5 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:51.841228 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507628a9-9a31-4521-8428-5683ae0d04f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:51.859918 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-507628a9-9a31-4521-8428-5683ae0d04f5 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:51.861395 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-507628a9-9a31-4521-8428-5683ae0d04f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:51.928368 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 62/133] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:42:51.967060 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932fbcc6-22d7-4531-9b03-a12785425ab6 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:52.072720 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932fbcc6-22d7-4531-9b03-a12785425ab6 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:52.105238 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f25153b-e202-48fc-b88d-3330200361d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:52.105858 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f25153b-e202-48fc-b88d-3330200361d2 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:52.106135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f25153b-e202-48fc-b88d-3330200361d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:52.106400 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f25153b-e202-48fc-b88d-3330200361d2 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:52.116814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f25153b-e202-48fc-b88d-3330200361d2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:42:52.121522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-4f25153b-e202-48fc-b88d-3330200361d2 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 30 00:42:52.121522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:52.121522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:42:52.121522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:42:52.121522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:42:52.121522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:42:52.121522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:42:52.121522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:42:52.121522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:42:52.121522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:42:52.121522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:42:52.121522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:42:52.121522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:42:52.121522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:42:52.121522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:42:52.121522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:42:52.122586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:42:52.122586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:42:52.122586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:42:52.122586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:42:52.122586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:42:52.122586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:42:52.122586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:42:52.122586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:42:52.122586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:42:52.122586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:42:52.122586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:42:52.122586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:42:52.122586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:42:52.122586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:42:52.122586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:42:52.122586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:42:52.123591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:42:52.123591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:42:52.123591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:42:52.123591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:42:52.123591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:42:52.123591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:42:52.123591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:42:52.123591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:42:52.123591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:42:52.123591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:42:52.123591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 30 00:42:52.123591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:42:52.123591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 72/134] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:42:51 2022] GET /identity/v3/roles/reader => generated 84 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:42:52.130506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507628a9-9a31-4521-8428-5683ae0d04f5 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:52.196767 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f25153b-e202-48fc-b88d-3330200361d2 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:52.230604 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:52.233768 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507628a9-9a31-4521-8428-5683ae0d04f5 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:52.239932 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:52.270125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7b7b85-af97-4750-9ff6-ceced0ebe1a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:52.270733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7b7b85-af97-4750-9ff6-ceced0ebe1a6 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:52.271008 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7b7b85-af97-4750-9ff6-ceced0ebe1a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:52.271277 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7b7b85-af97-4750-9ff6-ceced0ebe1a6 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:52.271966 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7b7b85-af97-4750-9ff6-ceced0ebe1a6 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:52.272264 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7b7b85-af97-4750-9ff6-ceced0ebe1a6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:52.273673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7b7b85-af97-4750-9ff6-ceced0ebe1a6 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:52.282018 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 63/135] 10.222.0.16 () {56 vars in 1101 bytes} [Sun Oct 30 00:42:52 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:52.287949 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7b7b85-af97-4750-9ff6-ceced0ebe1a6 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:52.297317 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f25153b-e202-48fc-b88d-3330200361d2 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:52.330833 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a97e86-2504-4244-899b-663d227f9a00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:52.331453 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a97e86-2504-4244-899b-663d227f9a00 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:52.331781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a97e86-2504-4244-899b-663d227f9a00 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:52.332047 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a97e86-2504-4244-899b-663d227f9a00 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:52.333286 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a97e86-2504-4244-899b-663d227f9a00 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:52.334735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a97e86-2504-4244-899b-663d227f9a00 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:52.336374 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 73/136] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:42:52 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:42:52.341737 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a97e86-2504-4244-899b-663d227f9a00 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:52.388611 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7b7b85-af97-4750-9ff6-ceced0ebe1a6 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:52.420655 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716dc436-83e1-498b-bf12-2d3a835053c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:52.421277 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716dc436-83e1-498b-bf12-2d3a835053c7 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:52.421550 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716dc436-83e1-498b-bf12-2d3a835053c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:52.421823 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716dc436-83e1-498b-bf12-2d3a835053c7 None admin] PATH_INFO: `/v3/users/9afe9a402b7641458693ec64fcd77589` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:52.422927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716dc436-83e1-498b-bf12-2d3a835053c7 None admin] RBAC: Authorizing `identity:get_user(user_id=9afe9a402b7641458693ec64fcd77589)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:52.424388 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716dc436-83e1-498b-bf12-2d3a835053c7 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:52.426283 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 64/137] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:42:52 2022] GET /identity/v3/users/9afe9a402b7641458693ec64fcd77589 => generated 289 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:52.431599 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716dc436-83e1-498b-bf12-2d3a835053c7 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:52.446522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a97e86-2504-4244-899b-663d227f9a00 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:52.456899 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:52.461734 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:52.481360 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dabc003-11fb-409e-ad92-f2230090d34c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:52.481965 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dabc003-11fb-409e-ad92-f2230090d34c None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:52.482239 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dabc003-11fb-409e-ad92-f2230090d34c None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:52.482506 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dabc003-11fb-409e-ad92-f2230090d34c None admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:52.483151 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dabc003-11fb-409e-ad92-f2230090d34c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:52.484627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dabc003-11fb-409e-ad92-f2230090d34c None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:52.512541 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 74/138] 10.222.0.16 () {60 vars in 1132 bytes} [Sun Oct 30 00:42:52 2022] POST /identity/v3/users => generated 283 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:42:52.539949 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716dc436-83e1-498b-bf12-2d3a835053c7 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:52.576132 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fbd2b2-9b5f-4d4e-a2a8-dc8ade2af811 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:52.576757 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fbd2b2-9b5f-4d4e-a2a8-dc8ade2af811 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:52.577034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fbd2b2-9b5f-4d4e-a2a8-dc8ade2af811 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:52.577303 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fbd2b2-9b5f-4d4e-a2a8-dc8ade2af811 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:52.652599 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09fbd2b2-9b5f-4d4e-a2a8-dc8ade2af811 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:52.654032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09fbd2b2-9b5f-4d4e-a2a8-dc8ade2af811 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:52.685262 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:52.759134 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 65/139] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:42:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 30 00:42:52.764879 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dabc003-11fb-409e-ad92-f2230090d34c None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:52.868716 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dabc003-11fb-409e-ad92-f2230090d34c None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:52.904465 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5458e2b9-e0ef-4710-831c-2c8d06c80c45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:52.905037 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5458e2b9-e0ef-4710-831c-2c8d06c80c45 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:52.905315 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5458e2b9-e0ef-4710-831c-2c8d06c80c45 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:52.905590 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5458e2b9-e0ef-4710-831c-2c8d06c80c45 None admin] PATH_INFO: `/v3/projects/e48719636cc1460283c3bc75e46b4a17` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:52.906745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5458e2b9-e0ef-4710-831c-2c8d06c80c45 None admin] RBAC: Authorizing `identity:get_project(project_id=e48719636cc1460283c3bc75e46b4a17)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:52.908237 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5458e2b9-e0ef-4710-831c-2c8d06c80c45 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:52.910094 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 75/140] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:42:52 2022] GET /identity/v3/projects/e48719636cc1460283c3bc75e46b4a17 => generated 298 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:42:52.916411 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fbd2b2-9b5f-4d4e-a2a8-dc8ade2af811 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:53.025287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fbd2b2-9b5f-4d4e-a2a8-dc8ade2af811 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:53.055069 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:53.061710 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cff056-6d14-45b1-8e30-0217599b58bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:53.062321 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cff056-6d14-45b1-8e30-0217599b58bd None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:53.062591 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cff056-6d14-45b1-8e30-0217599b58bd None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:53.062861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cff056-6d14-45b1-8e30-0217599b58bd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:53.063571 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9cff056-6d14-45b1-8e30-0217599b58bd None admin] RBAC: Adding query filter params (role.id=d330780cbb4e422991f4d18c0970c0b2, scope.project.id=e48719636cc1460283c3bc75e46b4a17, user.id=9afe9a402b7641458693ec64fcd77589) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:53.063902 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9cff056-6d14-45b1-8e30-0217599b58bd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:53.065342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9cff056-6d14-45b1-8e30-0217599b58bd None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:53.072120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 66/141] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:42:52 2022] GET /identity/v3/role_assignments?user.id=9afe9a402b7641458693ec64fcd77589&scope.project.id=e48719636cc1460283c3bc75e46b4a17&role.id=d330780cbb4e422991f4d18c0970c0b2 => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:54.757216 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02418251-c767-4b64-932e-6b61597595d2 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:54.757747 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02418251-c767-4b64-932e-6b61597595d2 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:54.757747 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02418251-c767-4b64-932e-6b61597595d2 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:54.759265 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 76/142] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:54.764320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232dca5f-2904-4f3d-8a1a-68d760eae731 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:54.764718 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232dca5f-2904-4f3d-8a1a-68d760eae731 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:54.764900 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232dca5f-2904-4f3d-8a1a-68d760eae731 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:54.783031 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-232dca5f-2904-4f3d-8a1a-68d760eae731 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:54.784235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-232dca5f-2904-4f3d-8a1a-68d760eae731 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:54.859530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 67/143] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:42:54.920769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e39a06-b6af-4d90-bad2-6a0c5f4cd56d None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:54.921091 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e39a06-b6af-4d90-bad2-6a0c5f4cd56d None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:54.921333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e39a06-b6af-4d90-bad2-6a0c5f4cd56d None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:54.922852 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 77/144] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:54.928319 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c3d440-f3b0-47a6-826b-34676c5c6551 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:54.928623 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c3d440-f3b0-47a6-826b-34676c5c6551 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:54.928900 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c3d440-f3b0-47a6-826b-34676c5c6551 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:54.948020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-97c3d440-f3b0-47a6-826b-34676c5c6551 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:54.949186 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-97c3d440-f3b0-47a6-826b-34676c5c6551 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:54.986052 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16986cf3-d393-440b-9963-66e722500413 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:54.986326 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16986cf3-d393-440b-9963-66e722500413 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:54.986599 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16986cf3-d393-440b-9963-66e722500413 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:54.988070 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 78/145] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:42:54.993383 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668401fd-db5d-4d2c-8f0b-d304948ce8a9 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:54.993747 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668401fd-db5d-4d2c-8f0b-d304948ce8a9 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:54.994116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668401fd-db5d-4d2c-8f0b-d304948ce8a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:55.012132 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-668401fd-db5d-4d2c-8f0b-d304948ce8a9 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:55.013363 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-668401fd-db5d-4d2c-8f0b-d304948ce8a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:55.019270 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 68/146] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:42:55.043250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:55.087704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 79/147] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:42:55.123909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c3d440-f3b0-47a6-826b-34676c5c6551 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:55.181415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:55.227163 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c3d440-f3b0-47a6-826b-34676c5c6551 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:55.260604 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03bbaf5a-3c86-4c3c-a131-c8a8d36d1d41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:55.261233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bbaf5a-3c86-4c3c-a131-c8a8d36d1d41 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:55.261508 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bbaf5a-3c86-4c3c-a131-c8a8d36d1d41 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:55.261782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bbaf5a-3c86-4c3c-a131-c8a8d36d1d41 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:55.272222 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03bbaf5a-3c86-4c3c-a131-c8a8d36d1d41 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:42:55.277000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-03bbaf5a-3c86-4c3c-a131-c8a8d36d1d41 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:42:55.277000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:55.277000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:42:55.277000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:42:55.277000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:42:55.277000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:42:55.277000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:42:55.277000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:42:55.277000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:42:55.277000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:42:55.277000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:42:55.277000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:42:55.277000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:42:55.277000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:42:55.277000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:42:55.277000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:42:55.277803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:42:55.277803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:42:55.277803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:42:55.277803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:42:55.277803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:42:55.277803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:42:55.277803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:42:55.277803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:42:55.277803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:42:55.277803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:42:55.277803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:42:55.277803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:42:55.277803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:42:55.277803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:42:55.277803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:42:55.277803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:42:55.278703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:42:55.278703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:42:55.278703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:42:55.278703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:42:55.278703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:42:55.278703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:42:55.278703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:42:55.278703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:42:55.278703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:42:55.278703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:42:55.278703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:42:55.278703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:42:55.278703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 69/148] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:42:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:42:55.283942 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668401fd-db5d-4d2c-8f0b-d304948ce8a9 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:55.284675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03bbaf5a-3c86-4c3c-a131-c8a8d36d1d41 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:55.389875 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668401fd-db5d-4d2c-8f0b-d304948ce8a9 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:55.390831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03bbaf5a-3c86-4c3c-a131-c8a8d36d1d41 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:55.405396 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:55.423991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d857c422-b11a-44a6-8bb9-ae3ff1a1ca80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:55.424602 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d857c422-b11a-44a6-8bb9-ae3ff1a1ca80 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:55.424878 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d857c422-b11a-44a6-8bb9-ae3ff1a1ca80 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:55.425147 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d857c422-b11a-44a6-8bb9-ae3ff1a1ca80 None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:55.425901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d857c422-b11a-44a6-8bb9-ae3ff1a1ca80 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:55.426188 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4218d3-fbea-4fd4-b4b5-c47b3f60cc1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:55.426297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d857c422-b11a-44a6-8bb9-ae3ff1a1ca80 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:55.426815 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4218d3-fbea-4fd4-b4b5-c47b3f60cc1d None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:55.427089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4218d3-fbea-4fd4-b4b5-c47b3f60cc1d None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:55.427356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4218d3-fbea-4fd4-b4b5-c47b3f60cc1d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:55.427591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d857c422-b11a-44a6-8bb9-ae3ff1a1ca80 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:55.435629 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 80/149] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:42:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:55.436762 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d857c422-b11a-44a6-8bb9-ae3ff1a1ca80 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:55.437764 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4218d3-fbea-4fd4-b4b5-c47b3f60cc1d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:42:55.443730 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-ca4218d3-fbea-4fd4-b4b5-c47b3f60cc1d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 30 00:42:55.443730 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:55.443730 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:42:55.443730 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:42:55.443730 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:42:55.443730 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:42:55.443730 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:42:55.443730 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:42:55.443730 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:42:55.443730 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:42:55.443730 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:42:55.443730 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:42:55.443730 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:42:55.443730 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:42:55.443730 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:42:55.443730 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:42:55.444755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:42:55.444755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:42:55.444755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:42:55.444755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:42:55.444755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:42:55.444755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:42:55.444755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:42:55.444755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:42:55.444755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:42:55.444755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:42:55.444755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:42:55.444755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:42:55.444755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:42:55.444755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:42:55.444755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:42:55.444755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:42:55.445729 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:42:55.445729 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:42:55.445729 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:42:55.445729 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:42:55.445729 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:42:55.445729 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:42:55.445729 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:42:55.445729 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:42:55.445729 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:42:55.445729 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:42:55.445729 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 30 00:42:55.445729 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:42:55.445729 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 70/150] 10.222.0.16 () {56 vars in 1114 bytes} [Sun Oct 30 00:42:55 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:42:55.446590 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4218d3-fbea-4fd4-b4b5-c47b3f60cc1d None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:55.467303 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:55.539552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d857c422-b11a-44a6-8bb9-ae3ff1a1ca80 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:55.547107 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4218d3-fbea-4fd4-b4b5-c47b3f60cc1d None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:55.572283 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e443077-765b-4861-ad54-062c147170b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:55.572909 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e443077-765b-4861-ad54-062c147170b2 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:55.573183 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e443077-765b-4861-ad54-062c147170b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:55.573472 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e443077-765b-4861-ad54-062c147170b2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:55.574581 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e443077-765b-4861-ad54-062c147170b2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:55.576044 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e443077-765b-4861-ad54-062c147170b2 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:55.577632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 81/151] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:42:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:55.580004 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e443077-765b-4861-ad54-062c147170b2 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:55.583062 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7e874c-fd6b-4af7-95eb-e3f46108af73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:55.583809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7e874c-fd6b-4af7-95eb-e3f46108af73 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:55.584085 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7e874c-fd6b-4af7-95eb-e3f46108af73 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:55.584352 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7e874c-fd6b-4af7-95eb-e3f46108af73 None admin] PATH_INFO: `/v3/users/c8a519dc93b2416fabcbaa7a2b1ea41e` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:55.598231 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7e874c-fd6b-4af7-95eb-e3f46108af73 None admin] RBAC: Authorizing `identity:get_user(user_id=c8a519dc93b2416fabcbaa7a2b1ea41e)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:55.599744 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7e874c-fd6b-4af7-95eb-e3f46108af73 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:55.601344 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 71/152] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:42:55 2022] GET /identity/v3/users/c8a519dc93b2416fabcbaa7a2b1ea41e => generated 283 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:55.602077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7e874c-fd6b-4af7-95eb-e3f46108af73 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:55.631695 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:55.675735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:55.682531 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e443077-765b-4861-ad54-062c147170b2 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:55.702194 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7e874c-fd6b-4af7-95eb-e3f46108af73 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:55.715900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96bc8f4-4e5a-4ce4-aa5b-36ab3fe60bce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:55.716609 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96bc8f4-4e5a-4ce4-aa5b-36ab3fe60bce None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:55.716887 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96bc8f4-4e5a-4ce4-aa5b-36ab3fe60bce None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:55.717180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96bc8f4-4e5a-4ce4-aa5b-36ab3fe60bce None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:55.717836 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96bc8f4-4e5a-4ce4-aa5b-36ab3fe60bce None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:55.718131 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96bc8f4-4e5a-4ce4-aa5b-36ab3fe60bce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:55.719499 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96bc8f4-4e5a-4ce4-aa5b-36ab3fe60bce None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:55.727470 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 82/153] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:42:55 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:55.728326 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96bc8f4-4e5a-4ce4-aa5b-36ab3fe60bce None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:55.738459 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ef54a4-120f-4289-a491-0f4ad14ad633 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:55.739162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ef54a4-120f-4289-a491-0f4ad14ad633 None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:55.739427 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ef54a4-120f-4289-a491-0f4ad14ad633 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:55.739690 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ef54a4-120f-4289-a491-0f4ad14ad633 None admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:55.740373 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ef54a4-120f-4289-a491-0f4ad14ad633 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:55.741818 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ef54a4-120f-4289-a491-0f4ad14ad633 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:55.762744 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 72/154] 10.222.0.16 () {60 vars in 1132 bytes} [Sun Oct 30 00:42:55 2022] POST /identity/v3/users => generated 297 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:42:55.763331 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ef54a4-120f-4289-a491-0f4ad14ad633 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:55.840727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96bc8f4-4e5a-4ce4-aa5b-36ab3fe60bce None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:55.859370 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:55.874359 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ef54a4-120f-4289-a491-0f4ad14ad633 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:55.880593 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd9592e-6f9f-4bac-92c9-b25dee6e5dbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:55.881235 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd9592e-6f9f-4bac-92c9-b25dee6e5dbe None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:55.881522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd9592e-6f9f-4bac-92c9-b25dee6e5dbe None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:55.882312 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd9592e-6f9f-4bac-92c9-b25dee6e5dbe None admin] PATH_INFO: `/v3/projects/ab2a3d04642247e2895bcc3311a9a205` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:55.883485 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd9592e-6f9f-4bac-92c9-b25dee6e5dbe None admin] RBAC: Authorizing `identity:get_project(project_id=ab2a3d04642247e2895bcc3311a9a205)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:55.885709 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd9592e-6f9f-4bac-92c9-b25dee6e5dbe None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:55.887634 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 83/155] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:42:55 2022] GET /identity/v3/projects/ab2a3d04642247e2895bcc3311a9a205 => generated 302 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:55.893391 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd9592e-6f9f-4bac-92c9-b25dee6e5dbe None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:55.911139 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2233059-bc79-445f-8337-b95234f9020f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:55.911818 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2233059-bc79-445f-8337-b95234f9020f None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:55.912237 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2233059-bc79-445f-8337-b95234f9020f None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:55.913401 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2233059-bc79-445f-8337-b95234f9020f None admin] PATH_INFO: `/v3/users/c8a519dc93b2416fabcbaa7a2b1ea41e` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:55.914694 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2233059-bc79-445f-8337-b95234f9020f None admin] RBAC: Authorizing `identity:get_user(user_id=c8a519dc93b2416fabcbaa7a2b1ea41e)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:55.916215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2233059-bc79-445f-8337-b95234f9020f None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:55.922814 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 73/156] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:42:55 2022] GET /identity/v3/users/c8a519dc93b2416fabcbaa7a2b1ea41e => generated 283 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:55.923617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2233059-bc79-445f-8337-b95234f9020f None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:55.927899 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:56.014161 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd9592e-6f9f-4bac-92c9-b25dee6e5dbe None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:56.053083 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2233059-bc79-445f-8337-b95234f9020f None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:56.058521 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be86b99-5f7e-43f2-a08c-0584e53b01fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:56.059391 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be86b99-5f7e-43f2-a08c-0584e53b01fd None admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:56.059839 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be86b99-5f7e-43f2-a08c-0584e53b01fd None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:56.060465 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be86b99-5f7e-43f2-a08c-0584e53b01fd None admin] PATH_INFO: `/v3/projects/ab2a3d04642247e2895bcc3311a9a205/users/c8a519dc93b2416fabcbaa7a2b1ea41e/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:56.062954 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be86b99-5f7e-43f2-a08c-0584e53b01fd None admin] RBAC: Authorizing `identity:create_grant(project_id=ab2a3d04642247e2895bcc3311a9a205, user_id=c8a519dc93b2416fabcbaa7a2b1ea41e, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:56.064477 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be86b99-5f7e-43f2-a08c-0584e53b01fd None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:56.081196 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 84/157] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:42:55 2022] PUT /identity/v3/projects/ab2a3d04642247e2895bcc3311a9a205/users/c8a519dc93b2416fabcbaa7a2b1ea41e/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 00:42:56.095936 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a38ca8b-2108-4fce-a50a-c80658147af0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:56.096559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a38ca8b-2108-4fce-a50a-c80658147af0 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:56.096833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a38ca8b-2108-4fce-a50a-c80658147af0 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:56.097101 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a38ca8b-2108-4fce-a50a-c80658147af0 None admin] PATH_INFO: `/v3/projects/ab2a3d04642247e2895bcc3311a9a205` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:56.098832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a38ca8b-2108-4fce-a50a-c80658147af0 None admin] RBAC: Authorizing `identity:get_project(project_id=ab2a3d04642247e2895bcc3311a9a205)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:56.100464 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a38ca8b-2108-4fce-a50a-c80658147af0 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:56.102944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 74/158] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:42:55 2022] GET /identity/v3/projects/ab2a3d04642247e2895bcc3311a9a205 => generated 302 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:42:56.108475 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be86b99-5f7e-43f2-a08c-0584e53b01fd None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:56.207362 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:56.227285 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be86b99-5f7e-43f2-a08c-0584e53b01fd None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:56.271198 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87568eb2-bcd9-47a9-a292-80dea12d4c7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:56.272236 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87568eb2-bcd9-47a9-a292-80dea12d4c7e None admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:56.272512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87568eb2-bcd9-47a9-a292-80dea12d4c7e None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:56.272780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87568eb2-bcd9-47a9-a292-80dea12d4c7e None admin] PATH_INFO: `/v3/projects/ab2a3d04642247e2895bcc3311a9a205/users/c8a519dc93b2416fabcbaa7a2b1ea41e/roles/22e839a60f6f414abd42c577025184a2` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:56.275011 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87568eb2-bcd9-47a9-a292-80dea12d4c7e None admin] RBAC: Authorizing `identity:create_grant(project_id=ab2a3d04642247e2895bcc3311a9a205, user_id=c8a519dc93b2416fabcbaa7a2b1ea41e, role_id=22e839a60f6f414abd42c577025184a2)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:56.276483 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87568eb2-bcd9-47a9-a292-80dea12d4c7e None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:56.290583 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 85/159] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:42:56 2022] PUT /identity/v3/projects/ab2a3d04642247e2895bcc3311a9a205/users/c8a519dc93b2416fabcbaa7a2b1ea41e/roles/22e839a60f6f414abd42c577025184a2 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:42:58.070316 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf52458-eee7-433f-b1dd-bab2338ac9e1 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:58.070883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf52458-eee7-433f-b1dd-bab2338ac9e1 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:58.070883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf52458-eee7-433f-b1dd-bab2338ac9e1 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:58.072512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 75/160] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:58.078249 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adcec92-03c7-46f9-a7e6-6bc8244440f9 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:58.078697 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adcec92-03c7-46f9-a7e6-6bc8244440f9 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:58.078783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adcec92-03c7-46f9-a7e6-6bc8244440f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:58.098520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-9adcec92-03c7-46f9-a7e6-6bc8244440f9 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:58.099743 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-9adcec92-03c7-46f9-a7e6-6bc8244440f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:58.162300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a69d79-09bd-4c28-8365-d46d893e0d9e None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:58.162620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a69d79-09bd-4c28-8365-d46d893e0d9e None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:58.162843 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a69d79-09bd-4c28-8365-d46d893e0d9e None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:58.164493 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 76/161] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:42:58.170220 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8737b59d-c9f8-4ce6-90cf-70679887353e None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:58.170529 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8737b59d-c9f8-4ce6-90cf-70679887353e None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:58.170763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8737b59d-c9f8-4ce6-90cf-70679887353e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:58.177039 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 86/162] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:42:58.194282 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-8737b59d-c9f8-4ce6-90cf-70679887353e None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:58.195932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-8737b59d-c9f8-4ce6-90cf-70679887353e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:58.219630 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:58.271035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 77/163] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:42:58.289367 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbf9dbf-77d2-4edb-b754-b39441333d96 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:58.289996 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbf9dbf-77d2-4edb-b754-b39441333d96 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:58.290481 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbf9dbf-77d2-4edb-b754-b39441333d96 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:58.292463 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 87/164] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:42:58.298003 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7deb558-c3da-4ddb-addb-8368eebfb0e3 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:58.298497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7deb558-c3da-4ddb-addb-8368eebfb0e3 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:58.298959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7deb558-c3da-4ddb-addb-8368eebfb0e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:58.317701 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-b7deb558-c3da-4ddb-addb-8368eebfb0e3 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:58.319184 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-b7deb558-c3da-4ddb-addb-8368eebfb0e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:58.391287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 78/165] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:42:58.466312 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbf9dbf-77d2-4edb-b754-b39441333d96 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:58.474539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc31d6f-cfe4-476d-9da8-c1adb4427f90 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:58.474818 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc31d6f-cfe4-476d-9da8-c1adb4427f90 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:58.475070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc31d6f-cfe4-476d-9da8-c1adb4427f90 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:58.477037 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 79/166] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:42:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:42:58.483422 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c13669d-7d72-4a55-b9b4-265a101c29f7 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:58.484214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c13669d-7d72-4a55-b9b4-265a101c29f7 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:58.484486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c13669d-7d72-4a55-b9b4-265a101c29f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:58.505064 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-7c13669d-7d72-4a55-b9b4-265a101c29f7 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:42:58.506232 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-7c13669d-7d72-4a55-b9b4-265a101c29f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:42:58.553426 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:58.580963 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbf9dbf-77d2-4edb-b754-b39441333d96 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:58.583370 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 80/167] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:42:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:42:58.588623 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c13669d-7d72-4a55-b9b4-265a101c29f7 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:58.606982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:58.616992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a241e94-c7fc-4251-a455-21545073f088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:58.617746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a241e94-c7fc-4251-a455-21545073f088 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:58.618158 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a241e94-c7fc-4251-a455-21545073f088 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:58.618501 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a241e94-c7fc-4251-a455-21545073f088 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:58.629283 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a241e94-c7fc-4251-a455-21545073f088 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:42:58.634392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-2a241e94-c7fc-4251-a455-21545073f088 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:42:58.634392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:58.634392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:42:58.634392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:42:58.634392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:42:58.634392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:42:58.634392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:42:58.634392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:42:58.634392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:42:58.634392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:42:58.634392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:42:58.634392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:42:58.634392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:42:58.634392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:42:58.634392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:42:58.634392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:42:58.635552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:42:58.635552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:42:58.635552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:42:58.635552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:42:58.635552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:42:58.635552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:42:58.635552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:42:58.635552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:42:58.635552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:42:58.635552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:42:58.635552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:42:58.635552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:42:58.635552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:42:58.635552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:42:58.635552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:42:58.635552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:42:58.636591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:42:58.636591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:42:58.636591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:42:58.636591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:42:58.636591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:42:58.636591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:42:58.636591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:42:58.636591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:42:58.636591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:42:58.636591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:42:58.636591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:42:58.636591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:42:58.637785 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 88/168] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:42:58 2022] GET /identity/v3/roles/member => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:42:58.643784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a241e94-c7fc-4251-a455-21545073f088 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:58.709674 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c13669d-7d72-4a55-b9b4-265a101c29f7 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:58.745538 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09a8119-a747-4d9b-b2f3-3e9ddae5e95b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:58.746075 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a241e94-c7fc-4251-a455-21545073f088 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:58.746196 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09a8119-a747-4d9b-b2f3-3e9ddae5e95b None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:58.746436 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09a8119-a747-4d9b-b2f3-3e9ddae5e95b None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:58.746703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09a8119-a747-4d9b-b2f3-3e9ddae5e95b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:58.747870 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09a8119-a747-4d9b-b2f3-3e9ddae5e95b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:58.749315 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09a8119-a747-4d9b-b2f3-3e9ddae5e95b None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:58.755834 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 81/169] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:42:58 2022] GET /identity/v3/domains/default => generated 199 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:58.755834 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09a8119-a747-4d9b-b2f3-3e9ddae5e95b None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:58.786643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e770cfc-c43c-4294-966a-0c8d326c6ec9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:58.787244 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e770cfc-c43c-4294-966a-0c8d326c6ec9 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:58.787522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e770cfc-c43c-4294-966a-0c8d326c6ec9 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:58.787823 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e770cfc-c43c-4294-966a-0c8d326c6ec9 None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:58.788476 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e770cfc-c43c-4294-966a-0c8d326c6ec9 None admin] RBAC: Adding query filter params (name=member) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:58.788770 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e770cfc-c43c-4294-966a-0c8d326c6ec9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:58.790150 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e770cfc-c43c-4294-966a-0c8d326c6ec9 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:58.791522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:58.804859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 89/170] 10.222.0.16 () {56 vars in 1101 bytes} [Sun Oct 30 00:42:58 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:42:58.804859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e770cfc-c43c-4294-966a-0c8d326c6ec9 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:58.848774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:58.868823 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09a8119-a747-4d9b-b2f3-3e9ddae5e95b None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:58.901892 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e770cfc-c43c-4294-966a-0c8d326c6ec9 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:58.902800 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d591cde1-4e93-4f44-8d2b-56979cb1002e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:58.903422 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d591cde1-4e93-4f44-8d2b-56979cb1002e None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:58.903694 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d591cde1-4e93-4f44-8d2b-56979cb1002e None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:58.904003 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d591cde1-4e93-4f44-8d2b-56979cb1002e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:58.914337 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d591cde1-4e93-4f44-8d2b-56979cb1002e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:42:58.919116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-d591cde1-4e93-4f44-8d2b-56979cb1002e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:42:58.919116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:58.919116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:42:58.919116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:42:58.919116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:42:58.919116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:42:58.919116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:42:58.919116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:42:58.919116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:42:58.919116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:42:58.919116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:42:58.919116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:42:58.919116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:42:58.919116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:42:58.919116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:42:58.919116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:42:58.920034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:42:58.920034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:42:58.920034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:42:58.920034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:42:58.920034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:42:58.920034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:42:58.920034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:42:58.920034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:42:58.920034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:42:58.920034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:42:58.920034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:42:58.920034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:42:58.920034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:42:58.920034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:42:58.920034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:42:58.920034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:42:58.920833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:42:58.920833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:42:58.920833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:42:58.920833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:42:58.920833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:42:58.920833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:42:58.920833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:42:58.920833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:42:58.920833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:42:58.920833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:42:58.920833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:42:58.920833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:42:58.920833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 82/171] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:42:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:42:58.921488 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d591cde1-4e93-4f44-8d2b-56979cb1002e None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:58.938203 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e07ea6-45d7-4879-af1e-c3061eac0a87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:58.938808 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e07ea6-45d7-4879-af1e-c3061eac0a87 None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:58.939082 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e07ea6-45d7-4879-af1e-c3061eac0a87 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:58.939350 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e07ea6-45d7-4879-af1e-c3061eac0a87 None admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:58.940023 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e07ea6-45d7-4879-af1e-c3061eac0a87 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:58.941514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e07ea6-45d7-4879-af1e-c3061eac0a87 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:58.963349 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 90/172] 10.222.0.16 () {60 vars in 1132 bytes} [Sun Oct 30 00:42:58 2022] POST /identity/v3/users => generated 297 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:42:58.968866 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e07ea6-45d7-4879-af1e-c3061eac0a87 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:59.027356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d591cde1-4e93-4f44-8d2b-56979cb1002e None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:59.038602 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:59.074518 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb57889-0c28-4597-b669-a40a24aba8e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:59.075833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb57889-0c28-4597-b669-a40a24aba8e8 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:59.076109 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb57889-0c28-4597-b669-a40a24aba8e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:59.076428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb57889-0c28-4597-b669-a40a24aba8e8 None admin] PATH_INFO: `/v3/users/9f0ea0fd4ce74450bd28aee888ebdce7` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:59.089872 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:59.093843 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb57889-0c28-4597-b669-a40a24aba8e8 None admin] RBAC: Authorizing `identity:get_user(user_id=9f0ea0fd4ce74450bd28aee888ebdce7)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:59.094033 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e07ea6-45d7-4879-af1e-c3061eac0a87 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:59.095359 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb57889-0c28-4597-b669-a40a24aba8e8 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:59.097349 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 83/173] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:42:58 2022] GET /identity/v3/users/9f0ea0fd4ce74450bd28aee888ebdce7 => generated 297 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:59.098524 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb57889-0c28-4597-b669-a40a24aba8e8 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:59.134599 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca988fb-454d-40ee-b4ad-a1d6b6c8e048 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:59.135153 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca988fb-454d-40ee-b4ad-a1d6b6c8e048 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:59.135430 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca988fb-454d-40ee-b4ad-a1d6b6c8e048 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:59.135699 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca988fb-454d-40ee-b4ad-a1d6b6c8e048 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:59.151046 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca988fb-454d-40ee-b4ad-a1d6b6c8e048 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:42:59.156502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-2ca988fb-454d-40ee-b4ad-a1d6b6c8e048 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 30 00:42:59.156502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:42:59.156502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:42:59.156502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:42:59.156502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:42:59.156502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:42:59.156502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:42:59.156502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:42:59.156502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:42:59.156502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:42:59.156502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:42:59.156502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:42:59.156502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:42:59.156502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:42:59.156502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:42:59.156502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:42:59.157379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:42:59.157379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:42:59.157379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:42:59.157379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:42:59.157379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:42:59.157379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:42:59.157379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:42:59.157379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:42:59.157379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:42:59.157379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:42:59.157379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:42:59.157379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:42:59.157379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:42:59.157379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:42:59.157379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:42:59.157379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:42:59.158168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:42:59.158168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:42:59.158168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:42:59.158168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:42:59.158168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:42:59.158168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:42:59.158168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:42:59.158168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:42:59.158168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:42:59.158168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:42:59.158168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 30 00:42:59.158168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:42:59.158168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 91/174] 10.222.0.16 () {56 vars in 1114 bytes} [Sun Oct 30 00:42:58 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:42:59.159317 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca988fb-454d-40ee-b4ad-a1d6b6c8e048 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:59.217716 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb57889-0c28-4597-b669-a40a24aba8e8 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:59.266780 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e27e109-74bd-4880-984b-ac2cf729b945 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:59.267695 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e27e109-74bd-4880-984b-ac2cf729b945 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:59.268278 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e27e109-74bd-4880-984b-ac2cf729b945 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:59.268546 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e27e109-74bd-4880-984b-ac2cf729b945 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:59.269390 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e27e109-74bd-4880-984b-ac2cf729b945 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:59.269560 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e27e109-74bd-4880-984b-ac2cf729b945 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:59.270948 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e27e109-74bd-4880-984b-ac2cf729b945 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:59.280307 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 84/175] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:42:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:59.281006 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca988fb-454d-40ee-b4ad-a1d6b6c8e048 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:59.281196 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e27e109-74bd-4880-984b-ac2cf729b945 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:59.294848 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:59.317472 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0aaf764-415e-4aac-a2b3-6e3f30241fdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:59.318098 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0aaf764-415e-4aac-a2b3-6e3f30241fdd None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:59.318375 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0aaf764-415e-4aac-a2b3-6e3f30241fdd None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:59.318642 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0aaf764-415e-4aac-a2b3-6e3f30241fdd None admin] PATH_INFO: `/v3/projects/ab2a3d04642247e2895bcc3311a9a205` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:59.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0aaf764-415e-4aac-a2b3-6e3f30241fdd None admin] RBAC: Authorizing `identity:get_project(project_id=ab2a3d04642247e2895bcc3311a9a205)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:59.322906 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0aaf764-415e-4aac-a2b3-6e3f30241fdd None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:59.324873 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 92/176] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:42:59 2022] GET /identity/v3/projects/ab2a3d04642247e2895bcc3311a9a205 => generated 302 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:59.325837 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0aaf764-415e-4aac-a2b3-6e3f30241fdd None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:59.387062 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:59.414945 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e27e109-74bd-4880-984b-ac2cf729b945 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:59.454680 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92d3db6-1527-42c2-b6d7-fee63510be9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:59.455300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92d3db6-1527-42c2-b6d7-fee63510be9d None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:59.455592 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92d3db6-1527-42c2-b6d7-fee63510be9d None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:59.455871 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92d3db6-1527-42c2-b6d7-fee63510be9d None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:59.456643 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92d3db6-1527-42c2-b6d7-fee63510be9d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:42:59.456946 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92d3db6-1527-42c2-b6d7-fee63510be9d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:59.458454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0aaf764-415e-4aac-a2b3-6e3f30241fdd None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:59.458564 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92d3db6-1527-42c2-b6d7-fee63510be9d None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:59.469384 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 85/177] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:42:59 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:59.470098 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92d3db6-1527-42c2-b6d7-fee63510be9d None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:59.495302 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee10dae-537e-4f27-be3b-bde99339c84f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:59.496358 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee10dae-537e-4f27-be3b-bde99339c84f None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:59.496819 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee10dae-537e-4f27-be3b-bde99339c84f None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:59.497253 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee10dae-537e-4f27-be3b-bde99339c84f None admin] PATH_INFO: `/v3/users/c8a519dc93b2416fabcbaa7a2b1ea41e` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:59.498616 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee10dae-537e-4f27-be3b-bde99339c84f None admin] RBAC: Authorizing `identity:get_user(user_id=c8a519dc93b2416fabcbaa7a2b1ea41e)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:59.500323 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee10dae-537e-4f27-be3b-bde99339c84f None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:59.502145 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 93/178] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:42:59 2022] GET /identity/v3/users/c8a519dc93b2416fabcbaa7a2b1ea41e => generated 283 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:59.503117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee10dae-537e-4f27-be3b-bde99339c84f None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:59.559534 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:59.599178 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92d3db6-1527-42c2-b6d7-fee63510be9d None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:59.615944 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee10dae-537e-4f27-be3b-bde99339c84f None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:59.626190 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:59.639503 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4101c9b-5b05-43c0-a8c2-edacbcb2d95f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:59.640142 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4101c9b-5b05-43c0-a8c2-edacbcb2d95f None admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:59.640674 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4101c9b-5b05-43c0-a8c2-edacbcb2d95f None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:59.640961 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4101c9b-5b05-43c0-a8c2-edacbcb2d95f None admin] PATH_INFO: `/v3/projects/ab2a3d04642247e2895bcc3311a9a205/users/9f0ea0fd4ce74450bd28aee888ebdce7/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:59.643580 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4101c9b-5b05-43c0-a8c2-edacbcb2d95f None admin] RBAC: Authorizing `identity:create_grant(project_id=ab2a3d04642247e2895bcc3311a9a205, user_id=9f0ea0fd4ce74450bd28aee888ebdce7, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:59.645614 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4101c9b-5b05-43c0-a8c2-edacbcb2d95f None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:59.654120 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1664931c-4cbf-4267-a062-7d47b81b0b11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:59.654970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1664931c-4cbf-4267-a062-7d47b81b0b11 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:59.655254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1664931c-4cbf-4267-a062-7d47b81b0b11 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:59.655525 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1664931c-4cbf-4267-a062-7d47b81b0b11 None admin] PATH_INFO: `/v3/users/c8a519dc93b2416fabcbaa7a2b1ea41e` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:59.656751 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1664931c-4cbf-4267-a062-7d47b81b0b11 None admin] RBAC: Authorizing `identity:get_user(user_id=c8a519dc93b2416fabcbaa7a2b1ea41e)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:59.658196 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1664931c-4cbf-4267-a062-7d47b81b0b11 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:59.660184 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 94/179] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:42:59 2022] GET /identity/v3/users/c8a519dc93b2416fabcbaa7a2b1ea41e => generated 283 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:42:59.660897 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1664931c-4cbf-4267-a062-7d47b81b0b11 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:59.672873 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 86/180] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:42:59 2022] PUT /identity/v3/projects/ab2a3d04642247e2895bcc3311a9a205/users/9f0ea0fd4ce74450bd28aee888ebdce7/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:42:59.675086 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4101c9b-5b05-43c0-a8c2-edacbcb2d95f None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:42:59.799936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1664931c-4cbf-4267-a062-7d47b81b0b11 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:59.812532 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4101c9b-5b05-43c0-a8c2-edacbcb2d95f None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:42:59.840274 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:59.874798 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58aaa709-8077-4f09-98dd-afc7504f8f85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:59.875436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58aaa709-8077-4f09-98dd-afc7504f8f85 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:59.875704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58aaa709-8077-4f09-98dd-afc7504f8f85 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:59.875993 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58aaa709-8077-4f09-98dd-afc7504f8f85 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:59.882451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5574430-a5fe-4cd8-b9e1-8b64fd8d6e18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:42:59.883067 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5574430-a5fe-4cd8-b9e1-8b64fd8d6e18 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:42:59.883338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5574430-a5fe-4cd8-b9e1-8b64fd8d6e18 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:42:59.883605 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5574430-a5fe-4cd8-b9e1-8b64fd8d6e18 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:42:59.960091 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:42:59.979499 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58aaa709-8077-4f09-98dd-afc7504f8f85 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:59.981035 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58aaa709-8077-4f09-98dd-afc7504f8f85 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:42:59.985142 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5574430-a5fe-4cd8-b9e1-8b64fd8d6e18 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:42:59.986709 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5574430-a5fe-4cd8-b9e1-8b64fd8d6e18 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:00.161158 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 95/181] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:42:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 500 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 30 00:43:00.164584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 87/182] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:42:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 490 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 30 00:43:00.166486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58aaa709-8077-4f09-98dd-afc7504f8f85 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:00.170763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5574430-a5fe-4cd8-b9e1-8b64fd8d6e18 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:00.196477 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:00.310756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:00.333112 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58aaa709-8077-4f09-98dd-afc7504f8f85 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:00.343369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5574430-a5fe-4cd8-b9e1-8b64fd8d6e18 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:00.382050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549f2cad-64b7-4b88-b103-2261a68f329f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:00.382695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549f2cad-64b7-4b88-b103-2261a68f329f None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:00.382972 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549f2cad-64b7-4b88-b103-2261a68f329f None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:00.383237 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549f2cad-64b7-4b88-b103-2261a68f329f None admin] PATH_INFO: `/v3/projects/ab2a3d04642247e2895bcc3311a9a205` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:00.385032 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549f2cad-64b7-4b88-b103-2261a68f329f None admin] RBAC: Authorizing `identity:get_project(project_id=ab2a3d04642247e2895bcc3311a9a205)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:00.386489 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549f2cad-64b7-4b88-b103-2261a68f329f None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:00.388193 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 96/183] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:43:00 2022] GET /identity/v3/projects/ab2a3d04642247e2895bcc3311a9a205 => generated 302 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:00.394204 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549f2cad-64b7-4b88-b103-2261a68f329f None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:00.419627 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f8ffd1-7c75-48f8-b795-21b395c0d8a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:00.420388 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f8ffd1-7c75-48f8-b795-21b395c0d8a5 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:00.420667 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f8ffd1-7c75-48f8-b795-21b395c0d8a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:00.420937 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f8ffd1-7c75-48f8-b795-21b395c0d8a5 None admin] PATH_INFO: `/v3/projects/ab2a3d04642247e2895bcc3311a9a205` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:00.423334 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f8ffd1-7c75-48f8-b795-21b395c0d8a5 None admin] RBAC: Authorizing `identity:get_project(project_id=ab2a3d04642247e2895bcc3311a9a205)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:00.424839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f8ffd1-7c75-48f8-b795-21b395c0d8a5 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:00.426860 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 88/184] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:43:00 2022] GET /identity/v3/projects/ab2a3d04642247e2895bcc3311a9a205 => generated 302 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:00.435369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f8ffd1-7c75-48f8-b795-21b395c0d8a5 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:00.550655 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:00.580565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549f2cad-64b7-4b88-b103-2261a68f329f None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:00.609826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f8ffd1-7c75-48f8-b795-21b395c0d8a5 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:00.645803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa41ddca-9d85-42de-a8b8-54cb974e7f87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:00.647022 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa41ddca-9d85-42de-a8b8-54cb974e7f87 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:00.647483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa41ddca-9d85-42de-a8b8-54cb974e7f87 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:00.647981 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa41ddca-9d85-42de-a8b8-54cb974e7f87 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:00.649295 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa41ddca-9d85-42de-a8b8-54cb974e7f87 None admin] RBAC: Adding query filter params (role.id=22e839a60f6f414abd42c577025184a2, scope.project.id=ab2a3d04642247e2895bcc3311a9a205, user.id=c8a519dc93b2416fabcbaa7a2b1ea41e) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:00.650040 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa41ddca-9d85-42de-a8b8-54cb974e7f87 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:00.651673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa41ddca-9d85-42de-a8b8-54cb974e7f87 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:00.655435 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6862f1-2174-4083-b304-ada753d4128b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:00.657086 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6862f1-2174-4083-b304-ada753d4128b None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:00.657718 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6862f1-2174-4083-b304-ada753d4128b None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:00.659741 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 89/185] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:43:00 2022] GET /identity/v3/role_assignments?user.id=c8a519dc93b2416fabcbaa7a2b1ea41e&scope.project.id=ab2a3d04642247e2895bcc3311a9a205&role.id=22e839a60f6f414abd42c577025184a2 => generated 608 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:00.660243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6862f1-2174-4083-b304-ada753d4128b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:00.661170 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6862f1-2174-4083-b304-ada753d4128b None admin] RBAC: Adding query filter params (role.id=35f59686b5d84291a8d9baf19287eb5a, scope.project.id=ab2a3d04642247e2895bcc3311a9a205, user.id=c8a519dc93b2416fabcbaa7a2b1ea41e) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:00.661662 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6862f1-2174-4083-b304-ada753d4128b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:00.663327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6862f1-2174-4083-b304-ada753d4128b None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:00.670655 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 97/186] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:43:00 2022] GET /identity/v3/role_assignments?user.id=c8a519dc93b2416fabcbaa7a2b1ea41e&scope.project.id=ab2a3d04642247e2895bcc3311a9a205&role.id=35f59686b5d84291a8d9baf19287eb5a => generated 608 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:01.384563 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba03cce-2945-4013-9022-af7872dfaac8 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:01.385072 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba03cce-2945-4013-9022-af7872dfaac8 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:01.385146 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba03cce-2945-4013-9022-af7872dfaac8 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:01.386592 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 90/187] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:01.392043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ddeeb9-ecda-439c-8fd8-de1acb243afe None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:01.392430 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ddeeb9-ecda-439c-8fd8-de1acb243afe None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:01.392586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ddeeb9-ecda-439c-8fd8-de1acb243afe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:01.411401 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-89ddeeb9-ecda-439c-8fd8-de1acb243afe None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:01.412607 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-89ddeeb9-ecda-439c-8fd8-de1acb243afe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:01.464755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294cb79e-4b2f-43e0-8bef-0d9ad6574492 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:01.465020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294cb79e-4b2f-43e0-8bef-0d9ad6574492 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:01.465288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294cb79e-4b2f-43e0-8bef-0d9ad6574492 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:01.466639 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 91/188] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:43:01.471578 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1188b6f-150a-46c7-beba-c126ef321281 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:01.471908 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1188b6f-150a-46c7-beba-c126ef321281 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:01.472301 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1188b6f-150a-46c7-beba-c126ef321281 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:01.483809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 98/189] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:01.490463 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-a1188b6f-150a-46c7-beba-c126ef321281 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:01.491650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-a1188b6f-150a-46c7-beba-c126ef321281 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:01.502835 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:01.559216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 92/190] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:43:01.747469 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ddeeb9-ecda-439c-8fd8-de1acb243afe None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:01.826341 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:01.835507 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1188b6f-150a-46c7-beba-c126ef321281 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:01.854605 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ddeeb9-ecda-439c-8fd8-de1acb243afe None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:01.890051 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ab4046-520c-4acf-93a2-9c2da126d03e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:01.890683 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ab4046-520c-4acf-93a2-9c2da126d03e None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:01.890982 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ab4046-520c-4acf-93a2-9c2da126d03e None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:01.891275 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ab4046-520c-4acf-93a2-9c2da126d03e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:01.901803 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ab4046-520c-4acf-93a2-9c2da126d03e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:01.906579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-b2ab4046-520c-4acf-93a2-9c2da126d03e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 30 00:43:01.906579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:01.906579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:01.906579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:01.906579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:01.906579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:01.906579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:01.906579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:01.906579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:01.906579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:01.906579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:01.906579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:01.906579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:01.906579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:01.906579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:01.906579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:01.907454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:01.907454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:01.907454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:01.907454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:01.907454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:01.907454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:01.907454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:01.907454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:01.907454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:01.907454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:01.907454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:01.907454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:01.907454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:01.907454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:01.907454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:01.907454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:01.912725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:01.912725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:01.912725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:01.912725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:01.912725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:01.912725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:01.912725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:01.912725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:01.912725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:01.912725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:01.912725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 30 00:43:01.912725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:01.912725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 99/191] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:43:01 2022] GET /identity/v3/roles/reader => generated 84 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:01.916691 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ab4046-520c-4acf-93a2-9c2da126d03e None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:01.942361 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1188b6f-150a-46c7-beba-c126ef321281 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:01.976566 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d675e433-4851-4f58-97bf-caf025c500e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:01.977194 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d675e433-4851-4f58-97bf-caf025c500e8 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:01.977471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d675e433-4851-4f58-97bf-caf025c500e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:01.977731 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d675e433-4851-4f58-97bf-caf025c500e8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:01.978845 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d675e433-4851-4f58-97bf-caf025c500e8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:01.980325 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d675e433-4851-4f58-97bf-caf025c500e8 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:01.982053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 93/192] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:01 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:01.987954 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d675e433-4851-4f58-97bf-caf025c500e8 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:02.004916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:02.026247 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ab4046-520c-4acf-93a2-9c2da126d03e None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:02.065644 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640d0341-4f9c-4ea1-a168-9281175dc2f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:02.066311 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640d0341-4f9c-4ea1-a168-9281175dc2f0 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:02.066691 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640d0341-4f9c-4ea1-a168-9281175dc2f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:02.066957 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640d0341-4f9c-4ea1-a168-9281175dc2f0 None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:02.067704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640d0341-4f9c-4ea1-a168-9281175dc2f0 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:02.068031 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640d0341-4f9c-4ea1-a168-9281175dc2f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:02.069426 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640d0341-4f9c-4ea1-a168-9281175dc2f0 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:02.075508 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:02.079760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 100/193] 10.222.0.16 () {56 vars in 1101 bytes} [Sun Oct 30 00:43:01 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:02.084581 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640d0341-4f9c-4ea1-a168-9281175dc2f0 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:02.106608 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d675e433-4851-4f58-97bf-caf025c500e8 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:02.145531 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bfce3b-935c-426a-a114-f21c3c100f10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:02.146238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bfce3b-935c-426a-a114-f21c3c100f10 None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:02.146515 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bfce3b-935c-426a-a114-f21c3c100f10 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:02.146982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bfce3b-935c-426a-a114-f21c3c100f10 None admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:02.147546 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59bfce3b-935c-426a-a114-f21c3c100f10 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:02.149161 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59bfce3b-935c-426a-a114-f21c3c100f10 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:02.170916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 94/194] 10.222.0.16 () {60 vars in 1132 bytes} [Sun Oct 30 00:43:01 2022] POST /identity/v3/users => generated 293 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:43:02.172149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be16d15b-fa41-4b2a-bab0-97f89b1462ca None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:02.172424 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be16d15b-fa41-4b2a-bab0-97f89b1462ca None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:02.172687 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be16d15b-fa41-4b2a-bab0-97f89b1462ca None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:02.173875 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 95/195] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:02 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:02.179551 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b7cccb-dd15-48cd-97fd-069d6fd36af8 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:02.179866 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b7cccb-dd15-48cd-97fd-069d6fd36af8 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:02.180140 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b7cccb-dd15-48cd-97fd-069d6fd36af8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:02.198144 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640d0341-4f9c-4ea1-a168-9281175dc2f0 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:02.200391 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-e7b7cccb-dd15-48cd-97fd-069d6fd36af8 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:02.201613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e7b7cccb-dd15-48cd-97fd-069d6fd36af8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:02.234610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0c5ae0-d865-485e-8005-92df0d21baec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:02.235256 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0c5ae0-d865-485e-8005-92df0d21baec None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:02.235536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0c5ae0-d865-485e-8005-92df0d21baec None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:02.235834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0c5ae0-d865-485e-8005-92df0d21baec None admin] PATH_INFO: `/v3/users/bf7050c9630e49e3a686498c67e8249b` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:02.254503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0c5ae0-d865-485e-8005-92df0d21baec None admin] RBAC: Authorizing `identity:get_user(user_id=bf7050c9630e49e3a686498c67e8249b)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:02.256460 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0c5ae0-d865-485e-8005-92df0d21baec None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:02.258669 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 101/196] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:43:02 2022] GET /identity/v3/users/bf7050c9630e49e3a686498c67e8249b => generated 297 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:02.264488 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0c5ae0-d865-485e-8005-92df0d21baec None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:02.278236 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:02.287760 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 96/197] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:43:02.340337 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:02.391977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0c5ae0-d865-485e-8005-92df0d21baec None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:02.426737 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2611bd4b-413e-4549-9d1c-bc7f8e2a2f9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:02.427364 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2611bd4b-413e-4549-9d1c-bc7f8e2a2f9c None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:02.427626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2611bd4b-413e-4549-9d1c-bc7f8e2a2f9c None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:02.427923 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2611bd4b-413e-4549-9d1c-bc7f8e2a2f9c None admin] PATH_INFO: `/v3/projects/ab2a3d04642247e2895bcc3311a9a205` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:02.429036 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2611bd4b-413e-4549-9d1c-bc7f8e2a2f9c None admin] RBAC: Authorizing `identity:get_project(project_id=ab2a3d04642247e2895bcc3311a9a205)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:02.430485 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2611bd4b-413e-4549-9d1c-bc7f8e2a2f9c None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:02.432344 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 102/198] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:43:02 2022] GET /identity/v3/projects/ab2a3d04642247e2895bcc3311a9a205 => generated 302 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:02.438412 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b7cccb-dd15-48cd-97fd-069d6fd36af8 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:02.547169 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b7cccb-dd15-48cd-97fd-069d6fd36af8 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:02.549055 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2611bd4b-413e-4549-9d1c-bc7f8e2a2f9c None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:02.589751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79db318f-44b2-4f97-9cfc-8c730246dc37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:02.590412 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79db318f-44b2-4f97-9cfc-8c730246dc37 None admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:02.590686 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79db318f-44b2-4f97-9cfc-8c730246dc37 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:02.590947 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79db318f-44b2-4f97-9cfc-8c730246dc37 None admin] PATH_INFO: `/v3/projects/ab2a3d04642247e2895bcc3311a9a205/users/bf7050c9630e49e3a686498c67e8249b/roles/d330780cbb4e422991f4d18c0970c0b2` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:02.593026 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79db318f-44b2-4f97-9cfc-8c730246dc37 None admin] RBAC: Authorizing `identity:create_grant(project_id=ab2a3d04642247e2895bcc3311a9a205, user_id=bf7050c9630e49e3a686498c67e8249b, role_id=d330780cbb4e422991f4d18c0970c0b2)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:02.594465 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79db318f-44b2-4f97-9cfc-8c730246dc37 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:02.610114 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 97/199] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:43:02 2022] PUT /identity/v3/projects/ab2a3d04642247e2895bcc3311a9a205/users/bf7050c9630e49e3a686498c67e8249b/roles/d330780cbb4e422991f4d18c0970c0b2 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:43:02.667348 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2611bd4b-413e-4549-9d1c-bc7f8e2a2f9c None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:02.699228 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:02.709300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63783e7-8a29-410e-a9da-f82e4fc01704 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:02.709959 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63783e7-8a29-410e-a9da-f82e4fc01704 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:02.710235 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63783e7-8a29-410e-a9da-f82e4fc01704 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:02.710500 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63783e7-8a29-410e-a9da-f82e4fc01704 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:02.722850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63783e7-8a29-410e-a9da-f82e4fc01704 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:02.727784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-a63783e7-8a29-410e-a9da-f82e4fc01704 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:43:02.727784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:02.727784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:02.727784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:02.727784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:02.727784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:02.727784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:02.727784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:02.727784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:02.727784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:02.727784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:02.727784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:02.727784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:02.727784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:02.727784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:02.727784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:02.728671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:02.728671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:02.728671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:02.728671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:02.728671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:02.728671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:02.728671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:02.728671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:02.728671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:02.728671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:02.728671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:02.728671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:02.728671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:02.728671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:02.728671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:02.728671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:02.729452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:02.729452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:02.729452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:02.729452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:02.729452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:02.729452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:02.729452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:02.729452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:02.729452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:02.729452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:02.729452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:43:02.729452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:02.729452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 103/200] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:43:02 2022] GET /identity/v3/roles/member => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:43:02.734935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79db318f-44b2-4f97-9cfc-8c730246dc37 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:02.807524 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:02.851200 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79db318f-44b2-4f97-9cfc-8c730246dc37 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:02.888096 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a76c787-5688-465e-9977-8780c735c663 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:02.888714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a76c787-5688-465e-9977-8780c735c663 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:02.888981 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a76c787-5688-465e-9977-8780c735c663 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:02.889245 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a76c787-5688-465e-9977-8780c735c663 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:02.889915 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a76c787-5688-465e-9977-8780c735c663 None admin] RBAC: Adding query filter params (name=member) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:02.890207 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a76c787-5688-465e-9977-8780c735c663 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:02.891576 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a76c787-5688-465e-9977-8780c735c663 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:02.900669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 98/201] 10.222.0.16 () {56 vars in 1101 bytes} [Sun Oct 30 00:43:02 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:02.906473 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63783e7-8a29-410e-a9da-f82e4fc01704 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:03.009381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63783e7-8a29-410e-a9da-f82e4fc01704 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:03.044711 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc7e41f-d83e-49bc-8ea7-f8666cff76e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:03.045723 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc7e41f-d83e-49bc-8ea7-f8666cff76e6 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:03.046191 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc7e41f-d83e-49bc-8ea7-f8666cff76e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:03.046663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc7e41f-d83e-49bc-8ea7-f8666cff76e6 None admin] PATH_INFO: `/v3/users/9f0ea0fd4ce74450bd28aee888ebdce7` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:03.048029 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc7e41f-d83e-49bc-8ea7-f8666cff76e6 None admin] RBAC: Authorizing `identity:get_user(user_id=9f0ea0fd4ce74450bd28aee888ebdce7)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:03.049749 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc7e41f-d83e-49bc-8ea7-f8666cff76e6 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:03.051678 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 104/202] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:43:02 2022] GET /identity/v3/users/9f0ea0fd4ce74450bd28aee888ebdce7 => generated 297 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:03.061506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a76c787-5688-465e-9977-8780c735c663 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:03.160871 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a76c787-5688-465e-9977-8780c735c663 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:03.182907 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:03.195467 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cea1b7-22c8-4654-8112-f49a69aae1f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:03.196109 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cea1b7-22c8-4654-8112-f49a69aae1f3 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:03.196381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cea1b7-22c8-4654-8112-f49a69aae1f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:03.196645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cea1b7-22c8-4654-8112-f49a69aae1f3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:03.270423 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19cea1b7-22c8-4654-8112-f49a69aae1f3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:03.271976 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19cea1b7-22c8-4654-8112-f49a69aae1f3 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:03.372451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 99/203] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:43:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 30 00:43:03.380818 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc7e41f-d83e-49bc-8ea7-f8666cff76e6 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:03.436939 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:03.483663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc7e41f-d83e-49bc-8ea7-f8666cff76e6 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:03.520551 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ead2e2-68f7-488b-b1c0-1c557912a3c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:03.521199 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ead2e2-68f7-488b-b1c0-1c557912a3c8 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:03.521477 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ead2e2-68f7-488b-b1c0-1c557912a3c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:03.521745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ead2e2-68f7-488b-b1c0-1c557912a3c8 None admin] PATH_INFO: `/v3/projects/ab2a3d04642247e2895bcc3311a9a205` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:03.522879 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ead2e2-68f7-488b-b1c0-1c557912a3c8 None admin] RBAC: Authorizing `identity:get_project(project_id=ab2a3d04642247e2895bcc3311a9a205)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:03.524361 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ead2e2-68f7-488b-b1c0-1c557912a3c8 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:03.526074 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 105/204] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:43:03 2022] GET /identity/v3/projects/ab2a3d04642247e2895bcc3311a9a205 => generated 302 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:03.532202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cea1b7-22c8-4654-8112-f49a69aae1f3 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:03.559808 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:03.640076 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cea1b7-22c8-4654-8112-f49a69aae1f3 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:03.675516 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851e1d7f-c818-42f3-88e8-8fb94c2b7a4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:03.676537 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851e1d7f-c818-42f3-88e8-8fb94c2b7a4f None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:03.677006 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851e1d7f-c818-42f3-88e8-8fb94c2b7a4f None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:03.677466 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851e1d7f-c818-42f3-88e8-8fb94c2b7a4f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:03.678390 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851e1d7f-c818-42f3-88e8-8fb94c2b7a4f None admin] RBAC: Adding query filter params (role.id=5c19ef44708c4d1f95f5f657c5c79361, scope.project.id=ab2a3d04642247e2895bcc3311a9a205, user.id=9f0ea0fd4ce74450bd28aee888ebdce7) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:03.678900 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851e1d7f-c818-42f3-88e8-8fb94c2b7a4f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:03.680551 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851e1d7f-c818-42f3-88e8-8fb94c2b7a4f None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:03.687241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 100/205] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:43:03 2022] GET /identity/v3/role_assignments?user.id=9f0ea0fd4ce74450bd28aee888ebdce7&scope.project.id=ab2a3d04642247e2895bcc3311a9a205&role.id=5c19ef44708c4d1f95f5f657c5c79361 => generated 608 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:04.498846 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f6f2e7-4b06-4d49-b48e-22a99b1bd27a None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:04.499388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f6f2e7-4b06-4d49-b48e-22a99b1bd27a None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:04.499388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f6f2e7-4b06-4d49-b48e-22a99b1bd27a None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:04.500752 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 106/206] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:04.506353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67170296-1196-46a1-b9b3-22c4ae60bfe2 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:04.506607 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67170296-1196-46a1-b9b3-22c4ae60bfe2 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:04.506911 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67170296-1196-46a1-b9b3-22c4ae60bfe2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:04.525527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-67170296-1196-46a1-b9b3-22c4ae60bfe2 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:04.526731 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-67170296-1196-46a1-b9b3-22c4ae60bfe2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:04.581711 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd033e49-b855-4f00-b224-71863cd1c479 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:04.582084 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd033e49-b855-4f00-b224-71863cd1c479 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:04.582351 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd033e49-b855-4f00-b224-71863cd1c479 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:04.583690 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 107/207] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:43:04.588667 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3171072-da97-4d2a-83ef-15167cc76cf8 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:04.588957 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3171072-da97-4d2a-83ef-15167cc76cf8 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:04.589225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3171072-da97-4d2a-83ef-15167cc76cf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:04.597161 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 101/208] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:04.608234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-c3171072-da97-4d2a-83ef-15167cc76cf8 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:04.609478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-c3171072-da97-4d2a-83ef-15167cc76cf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:04.675836 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 108/209] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:43:04.860528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67170296-1196-46a1-b9b3-22c4ae60bfe2 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:04.878446 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:04.885698 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50bcbe3-d0e7-4e8c-90f8-9d0fc84604a2 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:04.885960 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50bcbe3-d0e7-4e8c-90f8-9d0fc84604a2 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:04.886404 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50bcbe3-d0e7-4e8c-90f8-9d0fc84604a2 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:04.887757 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 109/210] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:04.892672 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ec289c-f315-4d89-8494-ad8cda55d031 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:04.892934 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ec289c-f315-4d89-8494-ad8cda55d031 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:04.893193 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ec289c-f315-4d89-8494-ad8cda55d031 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:04.912031 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-b6ec289c-f315-4d89-8494-ad8cda55d031 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:04.913192 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b6ec289c-f315-4d89-8494-ad8cda55d031 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:04.959357 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:04.964707 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67170296-1196-46a1-b9b3-22c4ae60bfe2 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:04.979520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 110/211] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:43:04.980385 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ec289c-f315-4d89-8494-ad8cda55d031 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:05.003004 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3509ed3-d55a-4e9f-bb88-2ac4fc9f51a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:05.003688 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3509ed3-d55a-4e9f-bb88-2ac4fc9f51a8 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:05.003986 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3509ed3-d55a-4e9f-bb88-2ac4fc9f51a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:05.004235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3509ed3-d55a-4e9f-bb88-2ac4fc9f51a8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:05.014024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3509ed3-d55a-4e9f-bb88-2ac4fc9f51a8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:05.018791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-b3509ed3-d55a-4e9f-bb88-2ac4fc9f51a8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:43:05.018791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:05.018791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:05.018791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:05.018791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:05.018791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:05.018791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:05.018791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:05.018791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:05.018791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:05.018791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:05.018791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:05.018791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:05.018791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:05.018791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:05.018791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:05.019806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:05.019806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:05.019806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:05.019806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:05.019806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:05.019806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:05.019806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:05.019806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:05.019806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:05.019806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:05.019806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:05.019806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:05.019806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:05.019806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:05.019806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:05.019806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:05.020583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:05.020583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:05.020583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:05.020583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:05.020583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:05.020583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:05.020583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:05.020583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:05.020583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:05.020583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:05.020583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:43:05.020583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:05.020583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 102/212] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:43:04 2022] GET /identity/v3/roles/member => generated 84 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:05.027181 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3509ed3-d55a-4e9f-bb88-2ac4fc9f51a8 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:05.082042 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ec289c-f315-4d89-8494-ad8cda55d031 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:05.107835 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:05.121306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfa62c2-6b27-4fe7-a419-1425ff628c6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:05.121989 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfa62c2-6b27-4fe7-a419-1425ff628c6e None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:05.122262 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfa62c2-6b27-4fe7-a419-1425ff628c6e None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:05.122772 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfa62c2-6b27-4fe7-a419-1425ff628c6e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:05.124229 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfa62c2-6b27-4fe7-a419-1425ff628c6e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:05.126179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfa62c2-6b27-4fe7-a419-1425ff628c6e None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:05.128702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 111/213] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:04 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:05.133436 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3509ed3-d55a-4e9f-bb88-2ac4fc9f51a8 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:05.134025 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfa62c2-6b27-4fe7-a419-1425ff628c6e None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:05.170597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d72045-41a9-4925-b833-d5092530ba82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:05.171199 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d72045-41a9-4925-b833-d5092530ba82 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:05.171472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d72045-41a9-4925-b833-d5092530ba82 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:05.171760 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d72045-41a9-4925-b833-d5092530ba82 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:05.172416 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d72045-41a9-4925-b833-d5092530ba82 None admin] RBAC: Adding query filter params (name=member) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:05.172710 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d72045-41a9-4925-b833-d5092530ba82 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:05.174528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d72045-41a9-4925-b833-d5092530ba82 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:05.183687 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 103/214] 10.222.0.16 () {56 vars in 1101 bytes} [Sun Oct 30 00:43:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:05.184873 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:05.189629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d72045-41a9-4925-b833-d5092530ba82 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:05.240662 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfa62c2-6b27-4fe7-a419-1425ff628c6e None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:05.274288 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c209ec7-a11b-4a8e-8f29-9ef276fba3b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:05.274889 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c209ec7-a11b-4a8e-8f29-9ef276fba3b7 None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:05.275156 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c209ec7-a11b-4a8e-8f29-9ef276fba3b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:05.275419 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c209ec7-a11b-4a8e-8f29-9ef276fba3b7 None admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:05.276105 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c209ec7-a11b-4a8e-8f29-9ef276fba3b7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:05.277555 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c209ec7-a11b-4a8e-8f29-9ef276fba3b7 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:05.292152 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d72045-41a9-4925-b833-d5092530ba82 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:05.299190 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 112/215] 10.222.0.16 () {60 vars in 1132 bytes} [Sun Oct 30 00:43:05 2022] POST /identity/v3/users => generated 293 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:43:05.300207 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c209ec7-a11b-4a8e-8f29-9ef276fba3b7 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:05.328967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293922ba-8020-429a-a6d2-f01c578b431b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:05.330358 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293922ba-8020-429a-a6d2-f01c578b431b None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:05.330885 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293922ba-8020-429a-a6d2-f01c578b431b None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:05.331353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293922ba-8020-429a-a6d2-f01c578b431b None admin] PATH_INFO: `/v3/users/1fc6b435237b44ab9f7a541d2fd046c3` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:05.341359 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:05.352229 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293922ba-8020-429a-a6d2-f01c578b431b None admin] RBAC: Authorizing `identity:get_user(user_id=1fc6b435237b44ab9f7a541d2fd046c3)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:05.353810 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293922ba-8020-429a-a6d2-f01c578b431b None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:05.360848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 104/216] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:43:05 2022] GET /identity/v3/users/1fc6b435237b44ab9f7a541d2fd046c3 => generated 293 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:05.362120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293922ba-8020-429a-a6d2-f01c578b431b None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:05.418916 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c209ec7-a11b-4a8e-8f29-9ef276fba3b7 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:05.427383 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:05.457293 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb7d256-306a-4353-ad76-3af8076c8f26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:05.458002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb7d256-306a-4353-ad76-3af8076c8f26 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:05.458256 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb7d256-306a-4353-ad76-3af8076c8f26 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:05.458520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb7d256-306a-4353-ad76-3af8076c8f26 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:05.469488 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb7d256-306a-4353-ad76-3af8076c8f26 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:05.474761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-fbb7d256-306a-4353-ad76-3af8076c8f26 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 30 00:43:05.474761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:05.474761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:05.474761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:05.474761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:05.474761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:05.474761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:05.474761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:05.474761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:05.474761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:05.474761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:05.474761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:05.474761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:05.474761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:05.474761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:05.474761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:05.475653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:05.475653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:05.475653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:05.475653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:05.475653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:05.475653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:05.475653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:05.475653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:05.475653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:05.475653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:05.475653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:05.475653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:05.475653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:05.475653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:05.475653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:05.475653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:05.476471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:05.476471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:05.476471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:05.476471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:05.476471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:05.476471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:05.476471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:05.476471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:05.476471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:05.476471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:05.476471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 30 00:43:05.476471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:05.476471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 113/217] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:43:05 2022] GET /identity/v3/roles/reader => generated 84 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:05.481799 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb7d256-306a-4353-ad76-3af8076c8f26 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:05.482785 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293922ba-8020-429a-a6d2-f01c578b431b None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:05.530945 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfe0e84-70ce-44a7-97e3-edbd52af0fcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:05.531887 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfe0e84-70ce-44a7-97e3-edbd52af0fcd None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:05.532295 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfe0e84-70ce-44a7-97e3-edbd52af0fcd None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:05.532560 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfe0e84-70ce-44a7-97e3-edbd52af0fcd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:05.533294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfe0e84-70ce-44a7-97e3-edbd52af0fcd None admin] RBAC: Adding query filter params (role.id=5c19ef44708c4d1f95f5f657c5c79361, user.id=1fc6b435237b44ab9f7a541d2fd046c3, scope.system=all) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:05.533593 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfe0e84-70ce-44a7-97e3-edbd52af0fcd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:05.535755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfe0e84-70ce-44a7-97e3-edbd52af0fcd None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:05.540973 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 105/218] 10.222.0.16 () {56 vars in 1308 bytes} [Sun Oct 30 00:43:05 2022] GET /identity/v3/role_assignments?user.id=1fc6b435237b44ab9f7a541d2fd046c3&scope.system=all&role.id=5c19ef44708c4d1f95f5f657c5c79361 => generated 226 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:05.599775 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb7d256-306a-4353-ad76-3af8076c8f26 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:05.642758 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d0408c-0bfb-4c84-bc14-f540597bed41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:05.643383 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d0408c-0bfb-4c84-bc14-f540597bed41 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:05.643652 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d0408c-0bfb-4c84-bc14-f540597bed41 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:05.644147 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d0408c-0bfb-4c84-bc14-f540597bed41 None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:05.644873 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d0408c-0bfb-4c84-bc14-f540597bed41 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:05.645172 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d0408c-0bfb-4c84-bc14-f540597bed41 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:05.646579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d0408c-0bfb-4c84-bc14-f540597bed41 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:05.656444 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 114/219] 10.222.0.16 () {56 vars in 1101 bytes} [Sun Oct 30 00:43:05 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:05.662117 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfe0e84-70ce-44a7-97e3-edbd52af0fcd None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:05.730098 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:05.770656 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfe0e84-70ce-44a7-97e3-edbd52af0fcd None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:05.804194 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da62e11-6e10-4ce4-a157-2fd7b6fad36e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:05.804820 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da62e11-6e10-4ce4-a157-2fd7b6fad36e None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:05.805091 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da62e11-6e10-4ce4-a157-2fd7b6fad36e None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:05.805353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da62e11-6e10-4ce4-a157-2fd7b6fad36e None admin] PATH_INFO: `/v3/users/bf7050c9630e49e3a686498c67e8249b` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:05.806469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da62e11-6e10-4ce4-a157-2fd7b6fad36e None admin] RBAC: Authorizing `identity:get_user(user_id=bf7050c9630e49e3a686498c67e8249b)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:05.807932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da62e11-6e10-4ce4-a157-2fd7b6fad36e None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:05.809494 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 106/220] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:43:05 2022] GET /identity/v3/users/bf7050c9630e49e3a686498c67e8249b => generated 297 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:05.815375 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d0408c-0bfb-4c84-bc14-f540597bed41 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:05.844008 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:05.916778 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d0408c-0bfb-4c84-bc14-f540597bed41 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:05.951017 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dea8e48-cdc6-4ef4-994f-b9d3ed8df3e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:05.951624 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dea8e48-cdc6-4ef4-994f-b9d3ed8df3e0 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:05.951939 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dea8e48-cdc6-4ef4-994f-b9d3ed8df3e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:05.952206 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dea8e48-cdc6-4ef4-994f-b9d3ed8df3e0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:06.020182 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dea8e48-cdc6-4ef4-994f-b9d3ed8df3e0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:06.021740 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dea8e48-cdc6-4ef4-994f-b9d3ed8df3e0 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:06.060201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:06.130801 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 115/221] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:43:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 30 00:43:06.136171 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da62e11-6e10-4ce4-a157-2fd7b6fad36e None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:06.233889 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da62e11-6e10-4ce4-a157-2fd7b6fad36e None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:06.261437 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:06.269810 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0970119-063a-4f58-9735-3031f35388b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:06.270501 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0970119-063a-4f58-9735-3031f35388b0 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:06.270711 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0970119-063a-4f58-9735-3031f35388b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:06.270973 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0970119-063a-4f58-9735-3031f35388b0 None admin] PATH_INFO: `/v3/projects/ab2a3d04642247e2895bcc3311a9a205` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:06.272118 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0970119-063a-4f58-9735-3031f35388b0 None admin] RBAC: Authorizing `identity:get_project(project_id=ab2a3d04642247e2895bcc3311a9a205)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:06.273623 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0970119-063a-4f58-9735-3031f35388b0 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:06.275556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 107/222] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:43:06 2022] GET /identity/v3/projects/ab2a3d04642247e2895bcc3311a9a205 => generated 302 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:06.281519 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dea8e48-cdc6-4ef4-994f-b9d3ed8df3e0 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:06.387402 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dea8e48-cdc6-4ef4-994f-b9d3ed8df3e0 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:06.416662 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:06.424795 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6218cd64-fab1-4d7b-8537-85f0cfec7e8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:06.425562 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6218cd64-fab1-4d7b-8537-85f0cfec7e8f None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:06.425846 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6218cd64-fab1-4d7b-8537-85f0cfec7e8f None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:06.426108 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6218cd64-fab1-4d7b-8537-85f0cfec7e8f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:06.426832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6218cd64-fab1-4d7b-8537-85f0cfec7e8f None admin] RBAC: Adding query filter params (role.id=d330780cbb4e422991f4d18c0970c0b2, scope.project.id=ab2a3d04642247e2895bcc3311a9a205, user.id=bf7050c9630e49e3a686498c67e8249b) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:06.427128 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6218cd64-fab1-4d7b-8537-85f0cfec7e8f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:06.428583 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6218cd64-fab1-4d7b-8537-85f0cfec7e8f None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:06.434746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 116/223] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:43:06 2022] GET /identity/v3/role_assignments?user.id=bf7050c9630e49e3a686498c67e8249b&scope.project.id=ab2a3d04642247e2895bcc3311a9a205&role.id=d330780cbb4e422991f4d18c0970c0b2 => generated 608 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:07.642886 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128099b4-2806-4a3f-af05-72d0d6090dc5 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:07.643414 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128099b4-2806-4a3f-af05-72d0d6090dc5 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:07.643414 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128099b4-2806-4a3f-af05-72d0d6090dc5 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:07.644884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 108/224] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:07.650982 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cbef44-e6fb-4497-ac71-0d460ca00b72 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:07.651528 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cbef44-e6fb-4497-ac71-0d460ca00b72 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:07.651612 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cbef44-e6fb-4497-ac71-0d460ca00b72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:07.670518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-f4cbef44-e6fb-4497-ac71-0d460ca00b72 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:07.671710 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-f4cbef44-e6fb-4497-ac71-0d460ca00b72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:07.739742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c8f7db-693a-4e83-b997-98a7e3fcec2f None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:07.740036 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c8f7db-693a-4e83-b997-98a7e3fcec2f None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:07.740286 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c8f7db-693a-4e83-b997-98a7e3fcec2f None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:07.740840 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 117/225] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:07.742068 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e6879a-90e0-489c-a16e-c4d81cf7a06f None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:07.742341 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e6879a-90e0-489c-a16e-c4d81cf7a06f None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:07.742605 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e6879a-90e0-489c-a16e-c4d81cf7a06f None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:07.742766 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 109/226] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:07.743839 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 118/227] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:07 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:07.748418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4904004f-f16d-4678-a1c6-ba6c8757b65c None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:07.748694 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4904004f-f16d-4678-a1c6-ba6c8757b65c None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:07.748957 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4904004f-f16d-4678-a1c6-ba6c8757b65c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:07.749217 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9c4f2c-4ced-420f-8f15-4d349e0c865f None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:07.749625 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9c4f2c-4ced-420f-8f15-4d349e0c865f None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:07.750023 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9c4f2c-4ced-420f-8f15-4d349e0c865f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:07.767231 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-4904004f-f16d-4678-a1c6-ba6c8757b65c None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:07.768417 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-4904004f-f16d-4678-a1c6-ba6c8757b65c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:07.768948 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-da9c4f2c-4ced-420f-8f15-4d349e0c865f None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:07.770172 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-da9c4f2c-4ced-420f-8f15-4d349e0c865f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:07.838525 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 110/228] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:43:07.847758 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 119/229] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:43:08.007459 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4904004f-f16d-4678-a1c6-ba6c8757b65c None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:08.102808 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9c4f2c-4ced-420f-8f15-4d349e0c865f None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:08.107467 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4904004f-f16d-4678-a1c6-ba6c8757b65c None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:08.132554 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:08.141662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347a5c85-4213-4c07-983b-29a165df614c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:08.142270 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347a5c85-4213-4c07-983b-29a165df614c None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:08.142536 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347a5c85-4213-4c07-983b-29a165df614c None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:08.142796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347a5c85-4213-4c07-983b-29a165df614c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:08.152522 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347a5c85-4213-4c07-983b-29a165df614c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:08.155695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:08.157600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-347a5c85-4213-4c07-983b-29a165df614c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 30 00:43:08.157600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:08.157600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:08.157600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:08.157600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:08.157600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:08.157600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:08.157600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:08.157600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:08.157600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:08.157600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:08.157600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:08.157600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:08.157600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:08.157600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:08.157600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:08.158613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:08.158613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:08.158613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:08.158613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:08.158613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:08.158613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:08.158613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:08.158613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:08.158613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:08.158613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:08.158613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:08.158613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:08.158613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:08.158613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:08.158613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:08.158613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:08.160527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:08.160527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:08.160527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:08.160527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:08.160527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:08.160527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:08.160527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:08.160527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:08.160527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:08.160527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:08.160527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 30 00:43:08.160527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:08.160527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 111/230] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:43:08 2022] GET /identity/v3/roles/reader => generated 84 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:08.160527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347a5c85-4213-4c07-983b-29a165df614c None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:08.206476 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9c4f2c-4ced-420f-8f15-4d349e0c865f None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:08.240794 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb742d21-2752-4c60-902b-9faf37cb6ff2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:08.241408 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb742d21-2752-4c60-902b-9faf37cb6ff2 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:08.241675 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb742d21-2752-4c60-902b-9faf37cb6ff2 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:08.241936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb742d21-2752-4c60-902b-9faf37cb6ff2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:08.243001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb742d21-2752-4c60-902b-9faf37cb6ff2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:08.244460 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb742d21-2752-4c60-902b-9faf37cb6ff2 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:08.246217 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 120/231] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:08.246754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb742d21-2752-4c60-902b-9faf37cb6ff2 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:08.262042 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347a5c85-4213-4c07-983b-29a165df614c None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:08.294664 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b303f1d7-a9e2-4fe3-b48e-0304db4ed4aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:08.295272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b303f1d7-a9e2-4fe3-b48e-0304db4ed4aa None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:08.295539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b303f1d7-a9e2-4fe3-b48e-0304db4ed4aa None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:08.295833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b303f1d7-a9e2-4fe3-b48e-0304db4ed4aa None admin] PATH_INFO: `/v3/roles/member` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:08.306171 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b303f1d7-a9e2-4fe3-b48e-0304db4ed4aa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:08.311267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-b303f1d7-a9e2-4fe3-b48e-0304db4ed4aa None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:43:08.311267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:08.311267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:08.311267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:08.311267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:08.311267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:08.311267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:08.311267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:08.311267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:08.311267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:08.311267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:08.311267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:08.311267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:08.311267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:08.311267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:08.311267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:08.312222 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:08.312222 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:08.312222 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:08.312222 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:08.312222 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:08.312222 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:08.312222 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:08.312222 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:08.312222 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:08.312222 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:08.312222 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:08.312222 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:08.312222 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:08.312222 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:08.312222 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:08.312222 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:08.313011 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:08.313011 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:08.313011 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:08.313011 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:08.313011 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:08.313011 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:08.313011 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:08.313011 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:08.313011 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:08.313011 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:08.313011 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:43:08.313011 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:08.313011 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 112/232] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:43:08 2022] GET /identity/v3/roles/member => generated 84 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:08.313661 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b303f1d7-a9e2-4fe3-b48e-0304db4ed4aa None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:08.347335 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb742d21-2752-4c60-902b-9faf37cb6ff2 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:08.359945 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:08.369771 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:08.384610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd7ed98-a680-4b76-9e6f-f13bea010073 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:08.385211 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd7ed98-a680-4b76-9e6f-f13bea010073 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:08.385480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd7ed98-a680-4b76-9e6f-f13bea010073 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:08.385744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd7ed98-a680-4b76-9e6f-f13bea010073 None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:08.386391 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd7ed98-a680-4b76-9e6f-f13bea010073 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:08.386682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd7ed98-a680-4b76-9e6f-f13bea010073 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:08.388061 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd7ed98-a680-4b76-9e6f-f13bea010073 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:08.396046 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 121/233] 10.222.0.16 () {56 vars in 1101 bytes} [Sun Oct 30 00:43:08 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:08.396655 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd7ed98-a680-4b76-9e6f-f13bea010073 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:08.413743 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b303f1d7-a9e2-4fe3-b48e-0304db4ed4aa None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:08.447267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cdbfee-7c3e-49dd-9b5a-babd56ca239c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:08.447901 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cdbfee-7c3e-49dd-9b5a-babd56ca239c None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:08.448162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cdbfee-7c3e-49dd-9b5a-babd56ca239c None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:08.448423 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cdbfee-7c3e-49dd-9b5a-babd56ca239c None admin] PATH_INFO: `/v3/groups` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:08.449062 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cdbfee-7c3e-49dd-9b5a-babd56ca239c None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:08.450506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cdbfee-7c3e-49dd-9b5a-babd56ca239c None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:08.467993 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 113/234] 10.222.0.16 () {60 vars in 1134 bytes} [Sun Oct 30 00:43:08 2022] POST /identity/v3/groups => generated 229 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:43:08.468788 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cdbfee-7c3e-49dd-9b5a-babd56ca239c None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:08.504001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd7ed98-a680-4b76-9e6f-f13bea010073 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:08.542287 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5666a32-a80a-4e35-9fc4-27a0006452d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:08.542901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5666a32-a80a-4e35-9fc4-27a0006452d7 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:08.543182 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5666a32-a80a-4e35-9fc4-27a0006452d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:08.543453 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5666a32-a80a-4e35-9fc4-27a0006452d7 None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:08.544194 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5666a32-a80a-4e35-9fc4-27a0006452d7 None admin] RBAC: Adding query filter params (name=member) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:08.544496 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5666a32-a80a-4e35-9fc4-27a0006452d7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:08.546012 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5666a32-a80a-4e35-9fc4-27a0006452d7 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:08.562744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 122/235] 10.222.0.16 () {56 vars in 1101 bytes} [Sun Oct 30 00:43:08 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:08.564286 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5666a32-a80a-4e35-9fc4-27a0006452d7 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:08.584157 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cdbfee-7c3e-49dd-9b5a-babd56ca239c None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:08.592163 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:08.622403 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387c4092-da13-4e32-825b-99a17d78b1da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:08.623046 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387c4092-da13-4e32-825b-99a17d78b1da None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:08.623318 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387c4092-da13-4e32-825b-99a17d78b1da None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:08.623579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387c4092-da13-4e32-825b-99a17d78b1da None admin] PATH_INFO: `/v3/users/f605a3db82e445c5b982791dba79b40e` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:08.625912 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:08.638484 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387c4092-da13-4e32-825b-99a17d78b1da None admin] RBAC: Authorizing `identity:get_user(user_id=f605a3db82e445c5b982791dba79b40e)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:08.639992 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387c4092-da13-4e32-825b-99a17d78b1da None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:08.641851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 114/236] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:43:08 2022] GET /identity/v3/users/f605a3db82e445c5b982791dba79b40e => generated 293 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:08.647986 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387c4092-da13-4e32-825b-99a17d78b1da None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:08.687080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5666a32-a80a-4e35-9fc4-27a0006452d7 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:08.725025 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec6ef20-daeb-4aca-a874-b03cbb1ab4a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:08.725666 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec6ef20-daeb-4aca-a874-b03cbb1ab4a6 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:08.725940 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec6ef20-daeb-4aca-a874-b03cbb1ab4a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:08.726209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec6ef20-daeb-4aca-a874-b03cbb1ab4a6 None admin] PATH_INFO: `/v3/users/1fc6b435237b44ab9f7a541d2fd046c3` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:08.727335 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec6ef20-daeb-4aca-a874-b03cbb1ab4a6 None admin] RBAC: Authorizing `identity:get_user(user_id=1fc6b435237b44ab9f7a541d2fd046c3)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:08.728835 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec6ef20-daeb-4aca-a874-b03cbb1ab4a6 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:08.730592 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 123/237] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:43:08 2022] GET /identity/v3/users/1fc6b435237b44ab9f7a541d2fd046c3 => generated 293 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:08.735941 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec6ef20-daeb-4aca-a874-b03cbb1ab4a6 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:08.762110 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387c4092-da13-4e32-825b-99a17d78b1da None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:08.798811 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d79915-762d-4c1b-9c48-18dd948f4b8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:08.799325 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d79915-762d-4c1b-9c48-18dd948f4b8a None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:08.799591 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d79915-762d-4c1b-9c48-18dd948f4b8a None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:08.799918 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d79915-762d-4c1b-9c48-18dd948f4b8a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:08.800617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d79915-762d-4c1b-9c48-18dd948f4b8a None admin] RBAC: Adding query filter params (role.id=d330780cbb4e422991f4d18c0970c0b2, user.id=f605a3db82e445c5b982791dba79b40e, scope.system=all) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:08.800908 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d79915-762d-4c1b-9c48-18dd948f4b8a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:08.802434 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d79915-762d-4c1b-9c48-18dd948f4b8a None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:08.807761 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 115/238] 10.222.0.16 () {56 vars in 1308 bytes} [Sun Oct 30 00:43:08 2022] GET /identity/v3/role_assignments?user.id=f605a3db82e445c5b982791dba79b40e&scope.system=all&role.id=d330780cbb4e422991f4d18c0970c0b2 => generated 226 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:08.840863 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec6ef20-daeb-4aca-a874-b03cbb1ab4a6 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:08.855778 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:08.882401 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a617a3-96f2-4cbe-8ad1-cceb7971926c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:08.883163 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a617a3-96f2-4cbe-8ad1-cceb7971926c None admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:08.883511 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a617a3-96f2-4cbe-8ad1-cceb7971926c None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:08.883887 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a617a3-96f2-4cbe-8ad1-cceb7971926c None admin] PATH_INFO: `/v3/system/users/1fc6b435237b44ab9f7a541d2fd046c3/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:08.885481 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a617a3-96f2-4cbe-8ad1-cceb7971926c None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=1fc6b435237b44ab9f7a541d2fd046c3, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:08.886955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a617a3-96f2-4cbe-8ad1-cceb7971926c None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:08.897605 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 124/239] 10.222.0.16 () {58 vars in 1334 bytes} [Sun Oct 30 00:43:08 2022] PUT /identity/v3/system/users/1fc6b435237b44ab9f7a541d2fd046c3/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 00:43:10.634836 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c538a53-60f9-41ff-bd1e-85a654ae8b92 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:10.635335 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c538a53-60f9-41ff-bd1e-85a654ae8b92 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:10.635335 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c538a53-60f9-41ff-bd1e-85a654ae8b92 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:10.636809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 116/240] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:10.642100 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420590ab-bfab-4300-91ca-80c8532d121d None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:10.642504 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420590ab-bfab-4300-91ca-80c8532d121d None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:10.642599 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420590ab-bfab-4300-91ca-80c8532d121d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:10.661924 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-420590ab-bfab-4300-91ca-80c8532d121d None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:10.663101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-420590ab-bfab-4300-91ca-80c8532d121d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:10.732038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 125/241] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:10.924659 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1f52b3-d816-4955-85ba-321f12618831 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:10.925065 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1f52b3-d816-4955-85ba-321f12618831 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:10.925322 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1f52b3-d816-4955-85ba-321f12618831 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:10.926939 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 117/242] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:10.932639 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d03cb40-985b-46f5-89b7-5fc352f89bfc None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:10.932985 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d03cb40-985b-46f5-89b7-5fc352f89bfc None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:10.933150 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d03cb40-985b-46f5-89b7-5fc352f89bfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:10.954512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-5d03cb40-985b-46f5-89b7-5fc352f89bfc None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:10.955679 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-5d03cb40-985b-46f5-89b7-5fc352f89bfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:10.996766 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1f52b3-d816-4955-85ba-321f12618831 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:11.032815 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:11.033826 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 126/243] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:11.041685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3114ce88-135d-4325-8b75-1103d93bbcc8 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:11.041964 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3114ce88-135d-4325-8b75-1103d93bbcc8 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:11.042236 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3114ce88-135d-4325-8b75-1103d93bbcc8 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:11.044390 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 127/244] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:11.049473 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1a97bb-70a6-4a5f-91e4-902c2a44a014 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:11.049772 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1a97bb-70a6-4a5f-91e4-902c2a44a014 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:11.050076 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1a97bb-70a6-4a5f-91e4-902c2a44a014 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:11.068375 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-ed1a97bb-70a6-4a5f-91e4-902c2a44a014 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:11.069547 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-ed1a97bb-70a6-4a5f-91e4-902c2a44a014 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:11.100823 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:11.113403 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1f52b3-d816-4955-85ba-321f12618831 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:11.147758 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 128/245] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:43:11.147948 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498078d6-58e0-493e-baf5-456834053a7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:11.148389 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498078d6-58e0-493e-baf5-456834053a7d None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:11.148656 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498078d6-58e0-493e-baf5-456834053a7d None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:11.148920 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498078d6-58e0-493e-baf5-456834053a7d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:11.150445 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498078d6-58e0-493e-baf5-456834053a7d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:11.152828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498078d6-58e0-493e-baf5-456834053a7d None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:11.155746 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 118/246] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:11.160603 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1a97bb-70a6-4a5f-91e4-902c2a44a014 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:11.259901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1a97bb-70a6-4a5f-91e4-902c2a44a014 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:11.293122 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edeb061f-e27b-4ca5-afbb-3d04b5fcb023 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:11.293725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeb061f-e27b-4ca5-afbb-3d04b5fcb023 None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:11.293998 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeb061f-e27b-4ca5-afbb-3d04b5fcb023 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:11.294262 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeb061f-e27b-4ca5-afbb-3d04b5fcb023 None admin] PATH_INFO: `/v3/groups` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:11.294880 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edeb061f-e27b-4ca5-afbb-3d04b5fcb023 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:11.296560 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edeb061f-e27b-4ca5-afbb-3d04b5fcb023 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:11.297547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498078d6-58e0-493e-baf5-456834053a7d None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:11.315439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 129/247] 10.222.0.16 () {60 vars in 1134 bytes} [Sun Oct 30 00:43:11 2022] POST /identity/v3/groups => generated 226 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:43:11.413602 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:11.415519 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edeb061f-e27b-4ca5-afbb-3d04b5fcb023 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:11.421008 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498078d6-58e0-493e-baf5-456834053a7d None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:11.448165 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:11.462444 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219753bd-f46d-42dc-81b7-8e875cdca8be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:11.463090 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219753bd-f46d-42dc-81b7-8e875cdca8be None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:11.463365 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219753bd-f46d-42dc-81b7-8e875cdca8be None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:11.463632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219753bd-f46d-42dc-81b7-8e875cdca8be None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:11.475382 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219753bd-f46d-42dc-81b7-8e875cdca8be None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:11.480250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-219753bd-f46d-42dc-81b7-8e875cdca8be None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 30 00:43:11.480250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:11.480250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:11.480250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:11.480250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:11.480250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:11.480250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:11.480250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:11.480250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:11.480250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:11.480250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:11.480250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:11.480250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:11.480250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:11.480250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:11.480250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:11.481125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:11.481125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:11.481125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:11.481125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:11.481125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:11.481125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:11.481125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:11.481125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:11.481125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:11.481125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:11.481125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:11.481125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:11.481125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:11.481125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:11.481125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:11.481125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:11.481935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:11.481935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:11.481935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:11.481935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:11.481935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:11.481935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:11.481935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:11.481935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:11.481935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:11.481935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:11.481935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 30 00:43:11.481935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:11.481935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 119/248] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:43:11 2022] GET /identity/v3/roles/reader => generated 84 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:43:11.488879 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219753bd-f46d-42dc-81b7-8e875cdca8be None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:11.536356 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edeb061f-e27b-4ca5-afbb-3d04b5fcb023 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:11.578518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e903454-ca71-4829-ba94-99a28eb2f5c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:11.579362 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e903454-ca71-4829-ba94-99a28eb2f5c6 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:11.579808 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e903454-ca71-4829-ba94-99a28eb2f5c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:11.580176 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e903454-ca71-4829-ba94-99a28eb2f5c6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:11.596921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e903454-ca71-4829-ba94-99a28eb2f5c6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:11.602297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-5e903454-ca71-4829-ba94-99a28eb2f5c6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:43:11.602297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:11.602297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:11.602297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:11.602297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:11.602297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:11.602297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:11.602297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:11.602297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:11.602297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:11.602297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:11.602297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:11.602297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:11.602297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:11.602297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:11.602297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:11.603154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:11.603154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:11.603154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:11.603154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:11.603154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:11.603154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:11.603154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:11.603154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:11.603154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:11.603154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:11.603154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:11.603154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:11.603154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:11.603154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:11.603154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:11.603154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:11.603967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:11.603967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:11.603967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:11.603967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:11.603967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:11.603967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:11.603967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:11.603967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:11.603967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:11.603967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:11.603967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:43:11.603967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:11.603967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 130/249] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:43:11 2022] GET /identity/v3/roles/member => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:11.609472 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e903454-ca71-4829-ba94-99a28eb2f5c6 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:11.621554 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219753bd-f46d-42dc-81b7-8e875cdca8be None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:11.667871 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3112860b-323f-45c5-a29f-7472dd267372 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:11.669348 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3112860b-323f-45c5-a29f-7472dd267372 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:11.669850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3112860b-323f-45c5-a29f-7472dd267372 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:11.670114 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3112860b-323f-45c5-a29f-7472dd267372 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:11.670880 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3112860b-323f-45c5-a29f-7472dd267372 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:11.671177 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3112860b-323f-45c5-a29f-7472dd267372 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:11.672660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3112860b-323f-45c5-a29f-7472dd267372 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:11.685355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 120/250] 10.222.0.16 () {56 vars in 1101 bytes} [Sun Oct 30 00:43:11 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:11.691298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3112860b-323f-45c5-a29f-7472dd267372 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:11.717604 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:11.723893 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:11.747325 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e903454-ca71-4829-ba94-99a28eb2f5c6 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:11.784016 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6230aa-45db-43ce-b6ec-d270413cbc1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:11.784758 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6230aa-45db-43ce-b6ec-d270413cbc1b None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:11.785645 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6230aa-45db-43ce-b6ec-d270413cbc1b None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:11.786219 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6230aa-45db-43ce-b6ec-d270413cbc1b None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:11.786813 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6230aa-45db-43ce-b6ec-d270413cbc1b None admin] RBAC: Adding query filter params (name=member) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:11.787065 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6230aa-45db-43ce-b6ec-d270413cbc1b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:11.789241 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6230aa-45db-43ce-b6ec-d270413cbc1b None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:11.802516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 131/251] 10.222.0.16 () {56 vars in 1101 bytes} [Sun Oct 30 00:43:11 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:11.809398 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6230aa-45db-43ce-b6ec-d270413cbc1b None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:11.817579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3112860b-323f-45c5-a29f-7472dd267372 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:11.855882 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fd13d2-2bbb-4a6d-85e9-73e1a05e2486 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:11.856519 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fd13d2-2bbb-4a6d-85e9-73e1a05e2486 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:11.856795 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fd13d2-2bbb-4a6d-85e9-73e1a05e2486 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:11.857071 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fd13d2-2bbb-4a6d-85e9-73e1a05e2486 None admin] PATH_INFO: `/v3/users/f605a3db82e445c5b982791dba79b40e` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:11.858203 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6fd13d2-2bbb-4a6d-85e9-73e1a05e2486 None admin] RBAC: Authorizing `identity:get_user(user_id=f605a3db82e445c5b982791dba79b40e)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:11.859663 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6fd13d2-2bbb-4a6d-85e9-73e1a05e2486 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:11.861609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 121/252] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:43:11 2022] GET /identity/v3/users/f605a3db82e445c5b982791dba79b40e => generated 293 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:11.868744 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fd13d2-2bbb-4a6d-85e9-73e1a05e2486 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:11.928088 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6230aa-45db-43ce-b6ec-d270413cbc1b None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:11.974682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1449904f-904e-47e0-a73d-364d614fa956 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:11.974682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1449904f-904e-47e0-a73d-364d614fa956 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:11.974961 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1449904f-904e-47e0-a73d-364d614fa956 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:11.975362 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1449904f-904e-47e0-a73d-364d614fa956 None admin] PATH_INFO: `/v3/users/1fc6b435237b44ab9f7a541d2fd046c3` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:11.977866 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1449904f-904e-47e0-a73d-364d614fa956 None admin] RBAC: Authorizing `identity:get_user(user_id=1fc6b435237b44ab9f7a541d2fd046c3)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:11.978582 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1449904f-904e-47e0-a73d-364d614fa956 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:11.983816 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 132/253] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:43:11 2022] GET /identity/v3/users/1fc6b435237b44ab9f7a541d2fd046c3 => generated 293 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:11.988306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1449904f-904e-47e0-a73d-364d614fa956 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:11.989216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:12.017761 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fd13d2-2bbb-4a6d-85e9-73e1a05e2486 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:12.036075 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:12.060180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b132c129-9fda-4d18-8414-5b6cc5ac040e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:12.064200 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b132c129-9fda-4d18-8414-5b6cc5ac040e None admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:12.064706 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b132c129-9fda-4d18-8414-5b6cc5ac040e None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:12.065186 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b132c129-9fda-4d18-8414-5b6cc5ac040e None admin] PATH_INFO: `/v3/system/users/f605a3db82e445c5b982791dba79b40e/roles/d330780cbb4e422991f4d18c0970c0b2` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:12.067321 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b132c129-9fda-4d18-8414-5b6cc5ac040e None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=f605a3db82e445c5b982791dba79b40e, role_id=d330780cbb4e422991f4d18c0970c0b2)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:12.068998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b132c129-9fda-4d18-8414-5b6cc5ac040e None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:12.084398 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 122/254] 10.222.0.16 () {58 vars in 1334 bytes} [Sun Oct 30 00:43:11 2022] PUT /identity/v3/system/users/f605a3db82e445c5b982791dba79b40e/roles/d330780cbb4e422991f4d18c0970c0b2 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:43:12.137941 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1449904f-904e-47e0-a73d-364d614fa956 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:12.178248 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6ea0f1-565e-4713-a441-070633009505 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:12.179305 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6ea0f1-565e-4713-a441-070633009505 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:12.179778 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6ea0f1-565e-4713-a441-070633009505 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:12.180259 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6ea0f1-565e-4713-a441-070633009505 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:12.181183 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6ea0f1-565e-4713-a441-070633009505 None admin] RBAC: Adding query filter params (role.id=5c19ef44708c4d1f95f5f657c5c79361, user.id=1fc6b435237b44ab9f7a541d2fd046c3, scope.system=all) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:12.181656 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6ea0f1-565e-4713-a441-070633009505 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:12.183351 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6ea0f1-565e-4713-a441-070633009505 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:12.200531 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 133/255] 10.222.0.16 () {56 vars in 1308 bytes} [Sun Oct 30 00:43:11 2022] GET /identity/v3/role_assignments?user.id=1fc6b435237b44ab9f7a541d2fd046c3&scope.system=all&role.id=5c19ef44708c4d1f95f5f657c5c79361 => generated 510 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:13.687328 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f2eb2c-72b1-44f1-a0a2-fde369f18d10 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:13.687866 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f2eb2c-72b1-44f1-a0a2-fde369f18d10 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:13.687866 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f2eb2c-72b1-44f1-a0a2-fde369f18d10 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:13.689494 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 123/256] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:13.694798 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920da3e2-3eb2-4550-8f4b-461eb4f06857 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:13.695229 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920da3e2-3eb2-4550-8f4b-461eb4f06857 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:13.695402 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920da3e2-3eb2-4550-8f4b-461eb4f06857 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:13.716777 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-920da3e2-3eb2-4550-8f4b-461eb4f06857 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:13.717956 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-920da3e2-3eb2-4550-8f4b-461eb4f06857 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:13.785399 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 134/257] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:13 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 30 00:43:14.118258 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cfd155-6f7d-4759-8c01-be4bd851694b None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:14.118640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cfd155-6f7d-4759-8c01-be4bd851694b None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:14.118854 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cfd155-6f7d-4759-8c01-be4bd851694b None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:14.120472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 124/258] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:14.125901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7740f9-c945-4777-9664-3524b5dc237f None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:14.126190 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7740f9-c945-4777-9664-3524b5dc237f None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:14.126460 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7740f9-c945-4777-9664-3524b5dc237f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:14.150020 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-7b7740f9-c945-4777-9664-3524b5dc237f None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:14.151225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-7b7740f9-c945-4777-9664-3524b5dc237f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:14.162634 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 135/259] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:14 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 30 00:43:14.358108 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cfd155-6f7d-4759-8c01-be4bd851694b None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:14.377510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cfd155-6f7d-4759-8c01-be4bd851694b None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:14.382252 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ade13b5-5f91-4cb9-9557-68f7a411ba75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:14.382935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ade13b5-5f91-4cb9-9557-68f7a411ba75 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:14.383211 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ade13b5-5f91-4cb9-9557-68f7a411ba75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:14.383480 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ade13b5-5f91-4cb9-9557-68f7a411ba75 admin admin] PATH_INFO: `/v3/roles/member` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:14.402466 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ade13b5-5f91-4cb9-9557-68f7a411ba75 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:14.407415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-2ade13b5-5f91-4cb9-9557-68f7a411ba75 admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:43:14.407415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:14.407415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:14.407415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:14.407415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:14.407415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:14.407415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:14.407415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:14.407415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:14.407415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:14.407415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:14.407415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:14.407415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:14.407415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:14.407415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:14.407415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:14.408313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:14.408313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:14.408313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:14.408313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:14.408313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:14.408313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:14.408313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:14.408313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:14.408313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:14.408313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:14.408313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:14.408313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:14.408313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:14.408313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:14.408313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:14.408313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:14.409168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:14.409168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:14.409168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:14.409168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:14.409168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:14.409168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:14.409168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:14.409168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:14.409168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:14.409168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:14.409168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:43:14.409168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:14.409168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 125/260] 10.222.0.16 () {56 vars in 1120 bytes} [Sun Oct 30 00:43:14 2022] GET /identity/v3/roles/member => generated 84 bytes in 51 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:14.416210 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7740f9-c945-4777-9664-3524b5dc237f None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:14.426790 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7740f9-c945-4777-9664-3524b5dc237f None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:14.434575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47c622d-de5d-4d88-aa77-a733bed33c10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:14.435187 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47c622d-de5d-4d88-aa77-a733bed33c10 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:14.435459 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47c622d-de5d-4d88-aa77-a733bed33c10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:14.436061 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47c622d-de5d-4d88-aa77-a733bed33c10 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:14.436740 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47c622d-de5d-4d88-aa77-a733bed33c10 admin admin] RBAC: Adding query filter params (name=member) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:14.437034 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47c622d-de5d-4d88-aa77-a733bed33c10 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:14.438979 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: WARNING py.warnings [None req-d47c622d-de5d-4d88-aa77-a733bed33c10 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 30 00:43:14.438979 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: warnings.warn(msg) Oct 30 00:43:14.438979 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: Oct 30 00:43:14.439906 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47c622d-de5d-4d88-aa77-a733bed33c10 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:14.451516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 136/261] 10.222.0.16 () {56 vars in 1122 bytes} [Sun Oct 30 00:43:14 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:14.457318 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ade13b5-5f91-4cb9-9557-68f7a411ba75 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:14.467296 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ade13b5-5f91-4cb9-9557-68f7a411ba75 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:14.474401 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12142a87-5aa3-4a2c-971a-eb010fe4f80e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:14.475034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12142a87-5aa3-4a2c-971a-eb010fe4f80e admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:14.475312 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12142a87-5aa3-4a2c-971a-eb010fe4f80e admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:14.475582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12142a87-5aa3-4a2c-971a-eb010fe4f80e admin admin] PATH_INFO: `/v3/groups/23e7e0b4e22f4a35ac36c35c239812fd` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:14.484314 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12142a87-5aa3-4a2c-971a-eb010fe4f80e admin admin] RBAC: Authorizing `identity:get_group(group_id=23e7e0b4e22f4a35ac36c35c239812fd)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:14.486086 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: WARNING py.warnings [None req-12142a87-5aa3-4a2c-971a-eb010fe4f80e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 30 00:43:14.486086 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: warnings.warn(msg) Oct 30 00:43:14.486086 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: Oct 30 00:43:14.487369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12142a87-5aa3-4a2c-971a-eb010fe4f80e admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:14.489199 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 126/262] 10.222.0.16 () {56 vars in 1201 bytes} [Sun Oct 30 00:43:14 2022] GET /identity/v3/groups/23e7e0b4e22f4a35ac36c35c239812fd => generated 226 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:14.495003 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47c622d-de5d-4d88-aa77-a733bed33c10 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:14.501816 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:14.506390 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47c622d-de5d-4d88-aa77-a733bed33c10 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:14.512177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08dd076c-d7b1-457e-a92b-36c5bb049250 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:14.512880 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dd076c-d7b1-457e-a92b-36c5bb049250 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:14.513154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dd076c-d7b1-457e-a92b-36c5bb049250 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:14.513484 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dd076c-d7b1-457e-a92b-36c5bb049250 admin admin] PATH_INFO: `/v3/projects/e48719636cc1460283c3bc75e46b4a17` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:14.516101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08dd076c-d7b1-457e-a92b-36c5bb049250 admin admin] RBAC: Authorizing `identity:get_project(project_id=e48719636cc1460283c3bc75e46b4a17)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:14.518425 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08dd076c-d7b1-457e-a92b-36c5bb049250 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:14.520835 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 137/263] 10.222.0.16 () {56 vars in 1207 bytes} [Sun Oct 30 00:43:14 2022] GET /identity/v3/projects/e48719636cc1460283c3bc75e46b4a17 => generated 298 bytes in 26 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:14.526970 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12142a87-5aa3-4a2c-971a-eb010fe4f80e admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:14.539586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12142a87-5aa3-4a2c-971a-eb010fe4f80e admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:14.546655 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb667fb2-6ba2-4384-be78-53f8af1e8974 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:14.547355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb667fb2-6ba2-4384-be78-53f8af1e8974 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:14.547637 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb667fb2-6ba2-4384-be78-53f8af1e8974 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:14.547936 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb667fb2-6ba2-4384-be78-53f8af1e8974 admin admin] PATH_INFO: `/v3/projects/e48719636cc1460283c3bc75e46b4a17/groups/23e7e0b4e22f4a35ac36c35c239812fd/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:14.550058 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb667fb2-6ba2-4384-be78-53f8af1e8974 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e48719636cc1460283c3bc75e46b4a17, group_id=23e7e0b4e22f4a35ac36c35c239812fd, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:14.551078 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: WARNING py.warnings [None req-cb667fb2-6ba2-4384-be78-53f8af1e8974 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 30 00:43:14.551078 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: warnings.warn(msg) Oct 30 00:43:14.551078 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: Oct 30 00:43:14.553157 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb667fb2-6ba2-4384-be78-53f8af1e8974 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:14.573860 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 127/264] 10.222.0.16 () {58 vars in 1463 bytes} [Sun Oct 30 00:43:14 2022] PUT /identity/v3/projects/e48719636cc1460283c3bc75e46b4a17/groups/23e7e0b4e22f4a35ac36c35c239812fd/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:43:14.822296 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37715c6f-c346-4659-910b-753dbf4c1a47 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:14.823036 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37715c6f-c346-4659-910b-753dbf4c1a47 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:14.823036 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37715c6f-c346-4659-910b-753dbf4c1a47 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:14.824343 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 138/265] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:14.830122 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b69af57-c7f7-4c44-96f7-94abd7cdc414 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:14.830768 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b69af57-c7f7-4c44-96f7-94abd7cdc414 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:14.830768 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b69af57-c7f7-4c44-96f7-94abd7cdc414 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:14.832063 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37715c6f-c346-4659-910b-753dbf4c1a47 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:14.857855 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-4b69af57-c7f7-4c44-96f7-94abd7cdc414 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:14.860374 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-4b69af57-c7f7-4c44-96f7-94abd7cdc414 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:14.946712 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37715c6f-c346-4659-910b-753dbf4c1a47 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:14.962158 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a32092f-0d70-46a1-8107-bf051f7f9221 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:14.963176 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a32092f-0d70-46a1-8107-bf051f7f9221 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:14.963735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a32092f-0d70-46a1-8107-bf051f7f9221 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:14.964301 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a32092f-0d70-46a1-8107-bf051f7f9221 admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:14.971742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 128/266] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:14 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 30 00:43:14.992764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a32092f-0d70-46a1-8107-bf051f7f9221 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:14.999530 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-8a32092f-0d70-46a1-8107-bf051f7f9221 admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 30 00:43:14.999530 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:14.999530 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:14.999530 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:14.999530 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:14.999530 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:14.999530 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:14.999530 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:14.999530 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:14.999530 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:14.999530 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:14.999530 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:14.999530 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:14.999530 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:14.999530 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:14.999530 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:15.000452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:15.000452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:15.000452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:15.000452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:15.000452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:15.000452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:15.000452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:15.000452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:15.000452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:15.000452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:15.000452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:15.000452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:15.000452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:15.000452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:15.000452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:15.000452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:15.001258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:15.001258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:15.001258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:15.001258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:15.001258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:15.001258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:15.001258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:15.001258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:15.001258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:15.001258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:15.001258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 30 00:43:15.001258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:15.004442 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 139/267] 10.222.0.16 () {56 vars in 1135 bytes} [Sun Oct 30 00:43:14 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:15.010495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b69af57-c7f7-4c44-96f7-94abd7cdc414 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:15.024101 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b69af57-c7f7-4c44-96f7-94abd7cdc414 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:15.033235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597712b7-a043-434a-a452-03ddbec15ef1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:15.033907 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597712b7-a043-434a-a452-03ddbec15ef1 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:15.034252 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597712b7-a043-434a-a452-03ddbec15ef1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:15.034525 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597712b7-a043-434a-a452-03ddbec15ef1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:15.035248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597712b7-a043-434a-a452-03ddbec15ef1 admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:15.035550 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597712b7-a043-434a-a452-03ddbec15ef1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:15.036735 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: WARNING py.warnings [None req-597712b7-a043-434a-a452-03ddbec15ef1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 30 00:43:15.036735 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: warnings.warn(msg) Oct 30 00:43:15.036735 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: Oct 30 00:43:15.037689 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597712b7-a043-434a-a452-03ddbec15ef1 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:15.051632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 129/268] 10.222.0.16 () {56 vars in 1132 bytes} [Sun Oct 30 00:43:15 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:15.057111 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a32092f-0d70-46a1-8107-bf051f7f9221 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:15.073683 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a32092f-0d70-46a1-8107-bf051f7f9221 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:15.079888 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfa7d9c-f7c6-42ed-97c5-3c07237f3aa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:15.080760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfa7d9c-f7c6-42ed-97c5-3c07237f3aa4 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:15.081286 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfa7d9c-f7c6-42ed-97c5-3c07237f3aa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:15.081828 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfa7d9c-f7c6-42ed-97c5-3c07237f3aa4 admin admin] PATH_INFO: `/v3/groups/23e7e0b4e22f4a35ac36c35c239812fd` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:15.083165 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfa7d9c-f7c6-42ed-97c5-3c07237f3aa4 admin admin] RBAC: Authorizing `identity:get_group(group_id=23e7e0b4e22f4a35ac36c35c239812fd)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:15.084405 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: WARNING py.warnings [None req-1dfa7d9c-f7c6-42ed-97c5-3c07237f3aa4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 30 00:43:15.084405 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: warnings.warn(msg) Oct 30 00:43:15.084405 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: Oct 30 00:43:15.085972 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfa7d9c-f7c6-42ed-97c5-3c07237f3aa4 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:15.088158 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 140/269] 10.222.0.16 () {56 vars in 1201 bytes} [Sun Oct 30 00:43:15 2022] GET /identity/v3/groups/23e7e0b4e22f4a35ac36c35c239812fd => generated 226 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:15.093908 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597712b7-a043-434a-a452-03ddbec15ef1 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:15.104740 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597712b7-a043-434a-a452-03ddbec15ef1 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:15.109438 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3ba4af-bc4e-4f6a-abcc-ea0449111754 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:15.110306 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3ba4af-bc4e-4f6a-abcc-ea0449111754 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:15.110776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3ba4af-bc4e-4f6a-abcc-ea0449111754 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:15.111239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3ba4af-bc4e-4f6a-abcc-ea0449111754 admin admin] PATH_INFO: `/v3/projects/e48719636cc1460283c3bc75e46b4a17` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:15.114801 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3ba4af-bc4e-4f6a-abcc-ea0449111754 admin admin] RBAC: Authorizing `identity:get_project(project_id=e48719636cc1460283c3bc75e46b4a17)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:15.117139 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3ba4af-bc4e-4f6a-abcc-ea0449111754 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:15.119282 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 130/270] 10.222.0.16 () {56 vars in 1207 bytes} [Sun Oct 30 00:43:15 2022] GET /identity/v3/projects/e48719636cc1460283c3bc75e46b4a17 => generated 298 bytes in 26 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:15.125305 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfa7d9c-f7c6-42ed-97c5-3c07237f3aa4 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:15.136605 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfa7d9c-f7c6-42ed-97c5-3c07237f3aa4 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:15.141458 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5e89ae-41ae-4fdc-afa1-cc2e98d30b02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:15.142368 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5e89ae-41ae-4fdc-afa1-cc2e98d30b02 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:15.142836 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5e89ae-41ae-4fdc-afa1-cc2e98d30b02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:15.143299 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5e89ae-41ae-4fdc-afa1-cc2e98d30b02 admin admin] PATH_INFO: `/v3/projects/e48719636cc1460283c3bc75e46b4a17/groups/23e7e0b4e22f4a35ac36c35c239812fd/roles/22e839a60f6f414abd42c577025184a2` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:15.146090 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5e89ae-41ae-4fdc-afa1-cc2e98d30b02 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e48719636cc1460283c3bc75e46b4a17, group_id=23e7e0b4e22f4a35ac36c35c239812fd, role_id=22e839a60f6f414abd42c577025184a2)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:15.147325 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: WARNING py.warnings [None req-1f5e89ae-41ae-4fdc-afa1-cc2e98d30b02 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 30 00:43:15.147325 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: warnings.warn(msg) Oct 30 00:43:15.147325 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: Oct 30 00:43:15.149655 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5e89ae-41ae-4fdc-afa1-cc2e98d30b02 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:15.170707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 141/271] 10.222.0.16 () {58 vars in 1463 bytes} [Sun Oct 30 00:43:15 2022] PUT /identity/v3/projects/e48719636cc1460283c3bc75e46b4a17/groups/23e7e0b4e22f4a35ac36c35c239812fd/roles/22e839a60f6f414abd42c577025184a2 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:43:15.526085 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3ba4af-bc4e-4f6a-abcc-ea0449111754 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:15.541202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3ba4af-bc4e-4f6a-abcc-ea0449111754 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:15.563804 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1efe103-067f-41be-95cd-7ee575ff300d None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:15.564413 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1efe103-067f-41be-95cd-7ee575ff300d None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:15.564899 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1efe103-067f-41be-95cd-7ee575ff300d None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:15.566633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 142/272] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:43:15.575622 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca598e40-ef37-4a99-a8f6-83230327bcf4 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:15.575934 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca598e40-ef37-4a99-a8f6-83230327bcf4 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:15.576197 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca598e40-ef37-4a99-a8f6-83230327bcf4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:15.598552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-ca598e40-ef37-4a99-a8f6-83230327bcf4 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:15.599663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-ca598e40-ef37-4a99-a8f6-83230327bcf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:15.621791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:15.637284 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7d61d0-b021-43b0-a79c-2bf4beafd604 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:15.637926 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7d61d0-b021-43b0-a79c-2bf4beafd604 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:15.638239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7d61d0-b021-43b0-a79c-2bf4beafd604 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:15.638515 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7d61d0-b021-43b0-a79c-2bf4beafd604 admin admin] PATH_INFO: `/v3/roles/member` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:15.659703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7d61d0-b021-43b0-a79c-2bf4beafd604 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:15.665009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-be7d61d0-b021-43b0-a79c-2bf4beafd604 admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:43:15.665009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:15.665009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:15.665009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:15.665009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:15.665009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:15.665009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:15.665009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:15.665009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:15.665009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:15.665009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:15.665009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:15.665009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:15.665009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:15.665009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:15.665009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:15.665883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:15.665883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:15.665883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:15.665883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:15.665883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:15.665883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:15.665883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:15.665883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:15.665883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:15.665883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:15.665883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:15.665883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:15.665883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:15.665883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:15.665883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:15.665883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:15.666684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:15.666684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:15.666684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:15.666684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:15.666684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:15.666684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:15.666684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:15.666684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:15.666684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:15.666684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:15.666684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:43:15.666684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:15.671909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 131/273] 10.222.0.16 () {56 vars in 1120 bytes} [Sun Oct 30 00:43:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:15.683491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7d61d0-b021-43b0-a79c-2bf4beafd604 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:15.700535 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7d61d0-b021-43b0-a79c-2bf4beafd604 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:15.705073 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 143/274] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:43:15.712413 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfaf581-4e14-408e-bf70-df0646d1f8f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:15.713044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfaf581-4e14-408e-bf70-df0646d1f8f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:15.713321 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfaf581-4e14-408e-bf70-df0646d1f8f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:15.713588 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfaf581-4e14-408e-bf70-df0646d1f8f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:15.714259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfaf581-4e14-408e-bf70-df0646d1f8f7 admin admin] RBAC: Adding query filter params (name=member) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:15.714556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfaf581-4e14-408e-bf70-df0646d1f8f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:15.716200 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfaf581-4e14-408e-bf70-df0646d1f8f7 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:15.726423 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 132/275] 10.222.0.16 () {56 vars in 1122 bytes} [Sun Oct 30 00:43:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:15.730946 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca598e40-ef37-4a99-a8f6-83230327bcf4 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:15.743905 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca598e40-ef37-4a99-a8f6-83230327bcf4 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:15.748299 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dee6853-ecad-4fee-a762-44c3ef625f72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:15.748933 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dee6853-ecad-4fee-a762-44c3ef625f72 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:15.749209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dee6853-ecad-4fee-a762-44c3ef625f72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:15.749524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dee6853-ecad-4fee-a762-44c3ef625f72 admin admin] PATH_INFO: `/v3/groups/23e7e0b4e22f4a35ac36c35c239812fd` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:15.750685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dee6853-ecad-4fee-a762-44c3ef625f72 admin admin] RBAC: Authorizing `identity:get_group(group_id=23e7e0b4e22f4a35ac36c35c239812fd)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:15.752871 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dee6853-ecad-4fee-a762-44c3ef625f72 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:15.754550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 144/276] 10.222.0.16 () {56 vars in 1201 bytes} [Sun Oct 30 00:43:15 2022] GET /identity/v3/groups/23e7e0b4e22f4a35ac36c35c239812fd => generated 226 bytes in 24 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:15.760328 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfaf581-4e14-408e-bf70-df0646d1f8f7 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:15.770954 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfaf581-4e14-408e-bf70-df0646d1f8f7 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:15.776061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4839c245-5665-4fd7-bed5-85a8c1adfddf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:15.776702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4839c245-5665-4fd7-bed5-85a8c1adfddf admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:15.777029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4839c245-5665-4fd7-bed5-85a8c1adfddf admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:15.777302 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4839c245-5665-4fd7-bed5-85a8c1adfddf admin admin] PATH_INFO: `/v3/projects/ab2a3d04642247e2895bcc3311a9a205` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:15.778475 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4839c245-5665-4fd7-bed5-85a8c1adfddf admin admin] RBAC: Authorizing `identity:get_project(project_id=ab2a3d04642247e2895bcc3311a9a205)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:15.780770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4839c245-5665-4fd7-bed5-85a8c1adfddf admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:15.782758 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 133/277] 10.222.0.16 () {56 vars in 1207 bytes} [Sun Oct 30 00:43:15 2022] GET /identity/v3/projects/ab2a3d04642247e2895bcc3311a9a205 => generated 302 bytes in 23 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:15.788837 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dee6853-ecad-4fee-a762-44c3ef625f72 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:15.799299 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dee6853-ecad-4fee-a762-44c3ef625f72 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:15.804078 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2138c0b-e4b0-44ac-8655-de13484c8528 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:15.804777 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2138c0b-e4b0-44ac-8655-de13484c8528 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:15.805051 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2138c0b-e4b0-44ac-8655-de13484c8528 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:15.805320 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2138c0b-e4b0-44ac-8655-de13484c8528 admin admin] PATH_INFO: `/v3/projects/ab2a3d04642247e2895bcc3311a9a205/groups/23e7e0b4e22f4a35ac36c35c239812fd/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:15.807319 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2138c0b-e4b0-44ac-8655-de13484c8528 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab2a3d04642247e2895bcc3311a9a205, group_id=23e7e0b4e22f4a35ac36c35c239812fd, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:15.810162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2138c0b-e4b0-44ac-8655-de13484c8528 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:15.826971 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 145/278] 10.222.0.16 () {58 vars in 1463 bytes} [Sun Oct 30 00:43:15 2022] PUT /identity/v3/projects/ab2a3d04642247e2895bcc3311a9a205/groups/23e7e0b4e22f4a35ac36c35c239812fd/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:43:16.047356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b2edd5-0d71-48f2-bdca-1266b023d084 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:16.047904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b2edd5-0d71-48f2-bdca-1266b023d084 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:16.047904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b2edd5-0d71-48f2-bdca-1266b023d084 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:16.049788 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 134/279] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:16.065876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b09dbde-7684-4965-9866-80a75ceb7de8 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:16.066276 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b09dbde-7684-4965-9866-80a75ceb7de8 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:16.066394 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b09dbde-7684-4965-9866-80a75ceb7de8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:16.090544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-9b09dbde-7684-4965-9866-80a75ceb7de8 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:16.091783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-9b09dbde-7684-4965-9866-80a75ceb7de8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:16.136213 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:16.164344 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 146/280] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:16 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 30 00:43:16.190496 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee55afc-11b0-4c86-a5e4-52baaed3a309 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:16.190828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee55afc-11b0-4c86-a5e4-52baaed3a309 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:16.191044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee55afc-11b0-4c86-a5e4-52baaed3a309 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:16.192422 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 135/281] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:16.197816 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de946b6-3e68-4546-9f7a-f4e09416a93a None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:16.198099 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de946b6-3e68-4546-9f7a-f4e09416a93a None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:16.198365 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de946b6-3e68-4546-9f7a-f4e09416a93a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:16.220566 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-1de946b6-3e68-4546-9f7a-f4e09416a93a None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:16.221990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-1de946b6-3e68-4546-9f7a-f4e09416a93a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:16.232027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 147/282] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:16 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 30 00:43:16.273543 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee55afc-11b0-4c86-a5e4-52baaed3a309 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:16.406819 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee55afc-11b0-4c86-a5e4-52baaed3a309 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:16.445416 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78622fad-811b-4ef4-9600-2a6a19f657b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:16.446050 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78622fad-811b-4ef4-9600-2a6a19f657b2 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:16.446324 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78622fad-811b-4ef4-9600-2a6a19f657b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:16.446586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78622fad-811b-4ef4-9600-2a6a19f657b2 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:16.464445 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78622fad-811b-4ef4-9600-2a6a19f657b2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:16.469214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-78622fad-811b-4ef4-9600-2a6a19f657b2 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 30 00:43:16.469214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:16.469214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:16.469214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:16.469214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:16.469214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:16.469214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:16.469214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:16.469214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:16.469214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:16.469214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:16.469214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:16.469214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:16.469214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:16.469214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:16.469214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:16.470097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:16.470097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:16.470097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:16.470097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:16.470097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:16.470097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:16.470097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:16.470097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:16.470097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:16.470097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:16.470097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:16.470097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:16.470097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:16.470097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:16.470097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:16.470097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:16.470888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:16.470888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:16.470888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:16.470888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:16.470888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:16.470888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:16.470888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:16.470888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:16.470888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:16.470888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:16.470888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 30 00:43:16.470888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:16.470888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 136/283] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:43:16 2022] GET /identity/v3/roles/reader => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:43:16.476516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de946b6-3e68-4546-9f7a-f4e09416a93a None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:16.536028 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78622fad-811b-4ef4-9600-2a6a19f657b2 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:16.559222 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78622fad-811b-4ef4-9600-2a6a19f657b2 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:16.568084 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75f5f9b-e86a-4355-94fe-c6f35828dc97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:16.568709 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75f5f9b-e86a-4355-94fe-c6f35828dc97 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:16.568975 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75f5f9b-e86a-4355-94fe-c6f35828dc97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:16.569239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75f5f9b-e86a-4355-94fe-c6f35828dc97 admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:16.593378 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75f5f9b-e86a-4355-94fe-c6f35828dc97 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:16.598320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-d75f5f9b-e86a-4355-94fe-c6f35828dc97 admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 30 00:43:16.598320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:16.598320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:16.598320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:16.598320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:16.598320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:16.598320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:16.598320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:16.598320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:16.598320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:16.598320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:16.598320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:16.598320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:16.598320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:16.598320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:16.598320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:16.600002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:16.600002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:16.600002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:16.600002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:16.600002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:16.600002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:16.600002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:16.600002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:16.600002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:16.600002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:16.600002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:16.600002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:16.600002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:16.600002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:16.600002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:16.600002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:16.600967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:16.600967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:16.600967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:16.600967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:16.600967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:16.600967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:16.600967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:16.600967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:16.600967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:16.600967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:16.600967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 30 00:43:16.600967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:16.600967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 137/284] 10.222.0.16 () {56 vars in 1135 bytes} [Sun Oct 30 00:43:16 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 64 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:43:16.613053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75f5f9b-e86a-4355-94fe-c6f35828dc97 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:16.639430 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75f5f9b-e86a-4355-94fe-c6f35828dc97 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:16.653330 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61b8299-ca30-4b87-adc0-5ecce28bc2f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:16.653985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61b8299-ca30-4b87-adc0-5ecce28bc2f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:16.654254 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61b8299-ca30-4b87-adc0-5ecce28bc2f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:16.654518 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61b8299-ca30-4b87-adc0-5ecce28bc2f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:16.655177 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61b8299-ca30-4b87-adc0-5ecce28bc2f7 admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:16.655467 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61b8299-ca30-4b87-adc0-5ecce28bc2f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:16.657090 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61b8299-ca30-4b87-adc0-5ecce28bc2f7 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:16.682327 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 138/285] 10.222.0.16 () {56 vars in 1132 bytes} [Sun Oct 30 00:43:16 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:16.688561 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de946b6-3e68-4546-9f7a-f4e09416a93a None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:16.719823 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61b8299-ca30-4b87-adc0-5ecce28bc2f7 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:16.729869 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61b8299-ca30-4b87-adc0-5ecce28bc2f7 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:16.737829 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066a6a3a-98bc-4fa8-b3a6-27005d49c48d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:16.738630 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066a6a3a-98bc-4fa8-b3a6-27005d49c48d admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:16.738901 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066a6a3a-98bc-4fa8-b3a6-27005d49c48d admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:16.739165 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066a6a3a-98bc-4fa8-b3a6-27005d49c48d admin admin] PATH_INFO: `/v3/groups/23e7e0b4e22f4a35ac36c35c239812fd` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:16.740525 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-066a6a3a-98bc-4fa8-b3a6-27005d49c48d admin admin] RBAC: Authorizing `identity:get_group(group_id=23e7e0b4e22f4a35ac36c35c239812fd)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:16.742402 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-066a6a3a-98bc-4fa8-b3a6-27005d49c48d admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:16.744109 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 139/286] 10.222.0.16 () {56 vars in 1201 bytes} [Sun Oct 30 00:43:16 2022] GET /identity/v3/groups/23e7e0b4e22f4a35ac36c35c239812fd => generated 226 bytes in 25 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:16.744716 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066a6a3a-98bc-4fa8-b3a6-27005d49c48d admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:16.746236 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370dce3b-79a6-43e4-8281-b2e1d25bb1ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:16.746841 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370dce3b-79a6-43e4-8281-b2e1d25bb1ec None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:16.747111 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370dce3b-79a6-43e4-8281-b2e1d25bb1ec None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:16.747370 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370dce3b-79a6-43e4-8281-b2e1d25bb1ec None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:16.748058 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370dce3b-79a6-43e4-8281-b2e1d25bb1ec None admin] RBAC: Adding query filter params (name=reader) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:16.748355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370dce3b-79a6-43e4-8281-b2e1d25bb1ec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:16.749715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370dce3b-79a6-43e4-8281-b2e1d25bb1ec None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:16.764673 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 148/287] 10.222.0.16 () {56 vars in 1101 bytes} [Sun Oct 30 00:43:16 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:16.765890 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370dce3b-79a6-43e4-8281-b2e1d25bb1ec None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:16.766653 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066a6a3a-98bc-4fa8-b3a6-27005d49c48d admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:16.774209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf11528e-d309-4c6a-961d-854a120527a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:16.775131 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf11528e-d309-4c6a-961d-854a120527a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:16.775595 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf11528e-d309-4c6a-961d-854a120527a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:16.776387 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf11528e-d309-4c6a-961d-854a120527a4 admin admin] PATH_INFO: `/v3/roles/admin` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:16.781639 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370dce3b-79a6-43e4-8281-b2e1d25bb1ec None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:16.786085 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638dbf59-aaab-4b84-afb2-1a287eeec553 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:16.786756 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638dbf59-aaab-4b84-afb2-1a287eeec553 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:16.787152 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638dbf59-aaab-4b84-afb2-1a287eeec553 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:16.787456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638dbf59-aaab-4b84-afb2-1a287eeec553 admin admin] PATH_INFO: `/v3/projects/ab2a3d04642247e2895bcc3311a9a205` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:16.788646 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638dbf59-aaab-4b84-afb2-1a287eeec553 admin admin] RBAC: Authorizing `identity:get_project(project_id=ab2a3d04642247e2895bcc3311a9a205)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:16.790610 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf11528e-d309-4c6a-961d-854a120527a4 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:16.790890 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638dbf59-aaab-4b84-afb2-1a287eeec553 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:16.792837 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 149/288] 10.222.0.16 () {56 vars in 1207 bytes} [Sun Oct 30 00:43:16 2022] GET /identity/v3/projects/ab2a3d04642247e2895bcc3311a9a205 => generated 302 bytes in 27 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:16.793479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638dbf59-aaab-4b84-afb2-1a287eeec553 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:16.801079 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:16.805298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-cf11528e-d309-4c6a-961d-854a120527a4 admin admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:43:16.805298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:16.805298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:16.805298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:16.805298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:16.805298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:16.805298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:16.805298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:16.805298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:16.805298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:16.805298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:16.805298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:16.805298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:16.805298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:16.805298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:16.805298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:16.806506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:16.806506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:16.806506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:16.806506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:16.806506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:16.806506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:16.806506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:16.806506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:16.806506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:16.806506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:16.806506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:16.806506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:16.806506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:16.806506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:16.806506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:16.806506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:16.807454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:16.807454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:16.807454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:16.807454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:16.807454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:16.807454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:16.807454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:16.807454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:16.807454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:16.807454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:16.807454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:43:16.807454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:16.809504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 140/289] 10.222.0.16 () {56 vars in 1117 bytes} [Sun Oct 30 00:43:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 65 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:16.810974 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf11528e-d309-4c6a-961d-854a120527a4 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:16.825335 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf11528e-d309-4c6a-961d-854a120527a4 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:16.831783 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b4d4c3-7529-471a-8741-5130615c4f2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:16.831783 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b4d4c3-7529-471a-8741-5130615c4f2a admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:16.831783 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b4d4c3-7529-471a-8741-5130615c4f2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:16.831783 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b4d4c3-7529-471a-8741-5130615c4f2a admin admin] PATH_INFO: `/v3/projects/ab2a3d04642247e2895bcc3311a9a205/groups/23e7e0b4e22f4a35ac36c35c239812fd/roles/22e839a60f6f414abd42c577025184a2` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:16.836352 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b4d4c3-7529-471a-8741-5130615c4f2a admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab2a3d04642247e2895bcc3311a9a205, group_id=23e7e0b4e22f4a35ac36c35c239812fd, role_id=22e839a60f6f414abd42c577025184a2)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:16.839193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b4d4c3-7529-471a-8741-5130615c4f2a admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:16.855864 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 141/290] 10.222.0.16 () {58 vars in 1463 bytes} [Sun Oct 30 00:43:16 2022] PUT /identity/v3/projects/ab2a3d04642247e2895bcc3311a9a205/groups/23e7e0b4e22f4a35ac36c35c239812fd/roles/22e839a60f6f414abd42c577025184a2 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:43:16.856754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b4d4c3-7529-471a-8741-5130615c4f2a admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:16.858247 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:16.931174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b4d4c3-7529-471a-8741-5130615c4f2a admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:16.950267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998cf929-0aa5-4671-a7ef-7a670304bbe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:16.952600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998cf929-0aa5-4671-a7ef-7a670304bbe0 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:16.953949 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998cf929-0aa5-4671-a7ef-7a670304bbe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:16.954381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998cf929-0aa5-4671-a7ef-7a670304bbe0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:16.956102 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998cf929-0aa5-4671-a7ef-7a670304bbe0 admin admin] RBAC: Adding query filter params (name=admin) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:16.957673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998cf929-0aa5-4671-a7ef-7a670304bbe0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:16.959461 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998cf929-0aa5-4671-a7ef-7a670304bbe0 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:16.989823 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 142/291] 10.222.0.16 () {56 vars in 1120 bytes} [Sun Oct 30 00:43:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:16.996753 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998cf929-0aa5-4671-a7ef-7a670304bbe0 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:17.012992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638dbf59-aaab-4b84-afb2-1a287eeec553 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:17.019705 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998cf929-0aa5-4671-a7ef-7a670304bbe0 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:17.033795 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c83ca86-eac9-405b-9b8f-ed60ef8eb7aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:17.034828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c83ca86-eac9-405b-9b8f-ed60ef8eb7aa admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:17.035289 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c83ca86-eac9-405b-9b8f-ed60ef8eb7aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:17.035684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c83ca86-eac9-405b-9b8f-ed60ef8eb7aa admin admin] PATH_INFO: `/v3/groups/0cb476b7bf89498e95bd5a2dfae11222` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:17.050367 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c83ca86-eac9-405b-9b8f-ed60ef8eb7aa admin admin] RBAC: Authorizing `identity:get_group(group_id=0cb476b7bf89498e95bd5a2dfae11222)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:17.054500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c83ca86-eac9-405b-9b8f-ed60ef8eb7aa admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:17.059082 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 143/292] 10.222.0.16 () {56 vars in 1201 bytes} [Sun Oct 30 00:43:16 2022] GET /identity/v3/groups/0cb476b7bf89498e95bd5a2dfae11222 => generated 229 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:17.067992 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c83ca86-eac9-405b-9b8f-ed60ef8eb7aa admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:17.087063 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c83ca86-eac9-405b-9b8f-ed60ef8eb7aa admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:17.101634 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b443e5f7-7c02-4960-82a5-70b745346c7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:17.102867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b443e5f7-7c02-4960-82a5-70b745346c7d admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:17.103933 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b443e5f7-7c02-4960-82a5-70b745346c7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:17.104351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b443e5f7-7c02-4960-82a5-70b745346c7d admin admin] PATH_INFO: `/v3/projects/3907972f9f6248debeeda8ecd254820e` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:17.105151 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5555abb4-fdcf-49cd-9b3c-c6a85f6f4ac8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:17.106323 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5555abb4-fdcf-49cd-9b3c-c6a85f6f4ac8 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:17.107352 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5555abb4-fdcf-49cd-9b3c-c6a85f6f4ac8 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:17.107908 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b443e5f7-7c02-4960-82a5-70b745346c7d admin admin] RBAC: Authorizing `identity:get_project(project_id=3907972f9f6248debeeda8ecd254820e)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:17.108116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5555abb4-fdcf-49cd-9b3c-c6a85f6f4ac8 None admin] PATH_INFO: `/v3/users/f605a3db82e445c5b982791dba79b40e` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:17.110539 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5555abb4-fdcf-49cd-9b3c-c6a85f6f4ac8 None admin] RBAC: Authorizing `identity:get_user(user_id=f605a3db82e445c5b982791dba79b40e)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:17.112869 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5555abb4-fdcf-49cd-9b3c-c6a85f6f4ac8 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:17.114643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 150/293] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:43:16 2022] GET /identity/v3/users/f605a3db82e445c5b982791dba79b40e => generated 293 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:17.114771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b443e5f7-7c02-4960-82a5-70b745346c7d admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:17.120374 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 144/294] 10.222.0.16 () {56 vars in 1207 bytes} [Sun Oct 30 00:43:17 2022] GET /identity/v3/projects/3907972f9f6248debeeda8ecd254820e => generated 344 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:17.121467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5555abb4-fdcf-49cd-9b3c-c6a85f6f4ac8 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:17.126680 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b443e5f7-7c02-4960-82a5-70b745346c7d admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:17.142623 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b443e5f7-7c02-4960-82a5-70b745346c7d admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:17.150182 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3089ab46-e689-48ec-bf21-3a59d3195742 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:17.151180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3089ab46-e689-48ec-bf21-3a59d3195742 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:17.152398 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3089ab46-e689-48ec-bf21-3a59d3195742 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:17.152975 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3089ab46-e689-48ec-bf21-3a59d3195742 admin admin] PATH_INFO: `/v3/projects/3907972f9f6248debeeda8ecd254820e/groups/0cb476b7bf89498e95bd5a2dfae11222/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:17.157102 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3089ab46-e689-48ec-bf21-3a59d3195742 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3907972f9f6248debeeda8ecd254820e, group_id=0cb476b7bf89498e95bd5a2dfae11222, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:17.162820 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3089ab46-e689-48ec-bf21-3a59d3195742 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:17.184205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 145/295] 10.222.0.16 () {58 vars in 1463 bytes} [Sun Oct 30 00:43:17 2022] PUT /identity/v3/projects/3907972f9f6248debeeda8ecd254820e/groups/0cb476b7bf89498e95bd5a2dfae11222/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 00:43:17.265620 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5555abb4-fdcf-49cd-9b3c-c6a85f6f4ac8 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:17.268556 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:17.309296 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59919a4-59a1-406b-8bb0-74bf5d6e3728 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:17.310184 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59919a4-59a1-406b-8bb0-74bf5d6e3728 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:17.310600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59919a4-59a1-406b-8bb0-74bf5d6e3728 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:17.311306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59919a4-59a1-406b-8bb0-74bf5d6e3728 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:17.312224 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59919a4-59a1-406b-8bb0-74bf5d6e3728 None admin] RBAC: Adding query filter params (role.id=d330780cbb4e422991f4d18c0970c0b2, user.id=f605a3db82e445c5b982791dba79b40e, scope.system=all) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:17.312710 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59919a4-59a1-406b-8bb0-74bf5d6e3728 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:17.314312 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59919a4-59a1-406b-8bb0-74bf5d6e3728 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:17.331931 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 151/296] 10.222.0.16 () {56 vars in 1308 bytes} [Sun Oct 30 00:43:17 2022] GET /identity/v3/role_assignments?user.id=f605a3db82e445c5b982791dba79b40e&scope.system=all&role.id=d330780cbb4e422991f4d18c0970c0b2 => generated 510 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:18.270581 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95aedb4-3ec7-4ee5-9338-a8df8250e464 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:18.271101 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95aedb4-3ec7-4ee5-9338-a8df8250e464 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:18.271402 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95aedb4-3ec7-4ee5-9338-a8df8250e464 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:18.273155 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 146/297] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:18.278589 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015e04ca-8aa5-42b2-a2fc-34530d24ac4e None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:18.279027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015e04ca-8aa5-42b2-a2fc-34530d24ac4e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:18.279199 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015e04ca-8aa5-42b2-a2fc-34530d24ac4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:18.309182 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-015e04ca-8aa5-42b2-a2fc-34530d24ac4e None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:18.310383 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-015e04ca-8aa5-42b2-a2fc-34530d24ac4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:18.396593 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 152/298] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:18.892407 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36510c4-70b8-40a1-85e3-927e439f2337 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:18.892829 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36510c4-70b8-40a1-85e3-927e439f2337 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:18.892939 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36510c4-70b8-40a1-85e3-927e439f2337 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:18.894702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 147/299] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:18.900108 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2baeec2f-0075-43b7-a17d-20e4748a0ab8 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:18.900393 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2baeec2f-0075-43b7-a17d-20e4748a0ab8 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:18.900665 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2baeec2f-0075-43b7-a17d-20e4748a0ab8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:18.919543 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-2baeec2f-0075-43b7-a17d-20e4748a0ab8 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:18.920759 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-2baeec2f-0075-43b7-a17d-20e4748a0ab8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:18.923660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36510c4-70b8-40a1-85e3-927e439f2337 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:19.057700 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 153/300] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:43:19.089777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36510c4-70b8-40a1-85e3-927e439f2337 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:19.096783 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:19.139784 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300752eb-fff8-435e-80a2-787e827f33f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:19.140497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300752eb-fff8-435e-80a2-787e827f33f5 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:19.140779 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300752eb-fff8-435e-80a2-787e827f33f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:19.141051 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300752eb-fff8-435e-80a2-787e827f33f5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:19.165475 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300752eb-fff8-435e-80a2-787e827f33f5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:19.169792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-300752eb-fff8-435e-80a2-787e827f33f5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:43:19.169792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:19.169792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:19.169792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:19.169792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:19.169792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:19.169792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:19.169792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:19.169792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:19.169792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:19.169792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:19.169792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:19.169792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:19.169792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:19.169792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:19.169792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:19.170675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:19.170675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:19.170675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:19.170675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:19.170675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:19.170675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:19.170675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:19.170675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:19.170675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:19.170675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:19.170675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:19.170675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:19.170675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:19.170675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:19.170675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:19.170675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:19.171468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:19.171468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:19.171468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:19.171468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:19.171468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:19.171468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:19.171468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:19.171468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:19.171468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:19.171468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:19.171468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:43:19.171468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:19.171468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 148/301] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:43:18 2022] GET /identity/v3/roles/member => generated 84 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:43:19.177480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2baeec2f-0075-43b7-a17d-20e4748a0ab8 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:19.210216 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:19.280050 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6f7025-9b71-4af9-bb3a-cf1786c7cf54 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:19.280548 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6f7025-9b71-4af9-bb3a-cf1786c7cf54 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:19.280949 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6f7025-9b71-4af9-bb3a-cf1786c7cf54 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:19.282565 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 149/302] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:43:19.288821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7ff45f-c82d-4574-bd02-d3cf09907c2c None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:19.289101 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7ff45f-c82d-4574-bd02-d3cf09907c2c None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:19.289376 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7ff45f-c82d-4574-bd02-d3cf09907c2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:19.304454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2baeec2f-0075-43b7-a17d-20e4748a0ab8 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:19.319408 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-0d7ff45f-c82d-4574-bd02-d3cf09907c2c None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:19.320635 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-0d7ff45f-c82d-4574-bd02-d3cf09907c2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:19.377007 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15549bd4-e977-41a9-8ada-4e0065eda8bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:19.377764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15549bd4-e977-41a9-8ada-4e0065eda8bf None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:19.378078 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15549bd4-e977-41a9-8ada-4e0065eda8bf None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:19.378352 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15549bd4-e977-41a9-8ada-4e0065eda8bf None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:19.379050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15549bd4-e977-41a9-8ada-4e0065eda8bf None admin] RBAC: Adding query filter params (name=member) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:19.379350 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15549bd4-e977-41a9-8ada-4e0065eda8bf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:19.380829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15549bd4-e977-41a9-8ada-4e0065eda8bf None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:19.392972 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 154/303] 10.222.0.16 () {56 vars in 1101 bytes} [Sun Oct 30 00:43:19 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:19.399186 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15549bd4-e977-41a9-8ada-4e0065eda8bf None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:19.509791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 150/304] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:43:19.511445 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde213c9-e1cc-4fcb-bdeb-10f3bf804823 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:19.511995 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde213c9-e1cc-4fcb-bdeb-10f3bf804823 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:19.512462 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde213c9-e1cc-4fcb-bdeb-10f3bf804823 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:19.513376 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:19.514211 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 151/305] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:19.525135 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066e7f27-05d0-4682-8071-a1847124e1fc None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:19.525425 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066e7f27-05d0-4682-8071-a1847124e1fc None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:19.525670 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066e7f27-05d0-4682-8071-a1847124e1fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:19.529785 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15549bd4-e977-41a9-8ada-4e0065eda8bf None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:19.552016 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-066e7f27-05d0-4682-8071-a1847124e1fc None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:19.553193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-066e7f27-05d0-4682-8071-a1847124e1fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:19.576481 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c80a774-9244-48ee-ba9f-524df2b15994 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:19.577104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c80a774-9244-48ee-ba9f-524df2b15994 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:19.577382 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c80a774-9244-48ee-ba9f-524df2b15994 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:19.577701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c80a774-9244-48ee-ba9f-524df2b15994 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:19.634493 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 152/306] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:43:19.635255 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066e7f27-05d0-4682-8071-a1847124e1fc None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:19.661419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:19.672947 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c80a774-9244-48ee-ba9f-524df2b15994 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:19.675928 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c80a774-9244-48ee-ba9f-524df2b15994 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:19.749181 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066e7f27-05d0-4682-8071-a1847124e1fc None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:19.787584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c739d0c-b086-48a0-806d-c915d8d62ae5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:19.788165 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c739d0c-b086-48a0-806d-c915d8d62ae5 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:19.788439 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c739d0c-b086-48a0-806d-c915d8d62ae5 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:19.788702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c739d0c-b086-48a0-806d-c915d8d62ae5 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:19.792274 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:19.804879 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c739d0c-b086-48a0-806d-c915d8d62ae5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:19.810017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-5c739d0c-b086-48a0-806d-c915d8d62ae5 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 30 00:43:19.810017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:19.810017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:19.810017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:19.810017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:19.810017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:19.810017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:19.810017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:19.810017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:19.810017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:19.810017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:19.810017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:19.810017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:19.810017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:19.810017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:19.810017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:19.811191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:19.811191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:19.811191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:19.811191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:19.811191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:19.811191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:19.811191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:19.811191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:19.811191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:19.811191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:19.811191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:19.811191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:19.811191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:19.811191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:19.811191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:19.811191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:19.812043 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:19.812043 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:19.812043 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:19.812043 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:19.812043 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:19.812043 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:19.812043 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:19.812043 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:19.812043 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:19.812043 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:19.812043 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 30 00:43:19.812043 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:19.815920 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 153/307] 10.222.0.16 () {56 vars in 1114 bytes} [Sun Oct 30 00:43:19 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:19.817958 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c739d0c-b086-48a0-806d-c915d8d62ae5 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:19.827308 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 155/308] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:43:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 30 00:43:19.829031 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c80a774-9244-48ee-ba9f-524df2b15994 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:19.939933 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:19.969330 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c739d0c-b086-48a0-806d-c915d8d62ae5 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:19.976370 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c80a774-9244-48ee-ba9f-524df2b15994 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:20.014222 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d546721-6a92-47d2-9577-45821ce84f2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:20.015249 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d546721-6a92-47d2-9577-45821ce84f2b None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:20.015525 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d546721-6a92-47d2-9577-45821ce84f2b None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:20.015888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d546721-6a92-47d2-9577-45821ce84f2b None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:20.016546 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d546721-6a92-47d2-9577-45821ce84f2b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:20.016836 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d546721-6a92-47d2-9577-45821ce84f2b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:20.018208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d546721-6a92-47d2-9577-45821ce84f2b None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:20.025301 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2092a1f0-2628-4127-a7bd-f5ab05797e93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:20.026007 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2092a1f0-2628-4127-a7bd-f5ab05797e93 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:20.026283 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2092a1f0-2628-4127-a7bd-f5ab05797e93 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:20.026547 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2092a1f0-2628-4127-a7bd-f5ab05797e93 None admin] PATH_INFO: `/v3/projects/e48719636cc1460283c3bc75e46b4a17` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:20.027669 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2092a1f0-2628-4127-a7bd-f5ab05797e93 None admin] RBAC: Authorizing `identity:get_project(project_id=e48719636cc1460283c3bc75e46b4a17)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:20.028045 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 154/309] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:43:19 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:20.028890 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d546721-6a92-47d2-9577-45821ce84f2b None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:20.029281 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2092a1f0-2628-4127-a7bd-f5ab05797e93 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:20.031077 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 156/310] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:43:19 2022] GET /identity/v3/projects/e48719636cc1460283c3bc75e46b4a17 => generated 298 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:20.036734 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2092a1f0-2628-4127-a7bd-f5ab05797e93 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:20.108591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:20.168370 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d546721-6a92-47d2-9577-45821ce84f2b None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:20.173760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2092a1f0-2628-4127-a7bd-f5ab05797e93 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:20.218029 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6fd135-1d2a-4ed3-96d7-48d4f69ae15d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:20.218669 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6fd135-1d2a-4ed3-96d7-48d4f69ae15d None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:20.218883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0308f9-9426-4d00-b95d-a9a7e6c8dc4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:20.218995 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6fd135-1d2a-4ed3-96d7-48d4f69ae15d None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:20.219200 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6fd135-1d2a-4ed3-96d7-48d4f69ae15d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:20.219569 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0308f9-9426-4d00-b95d-a9a7e6c8dc4b None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:20.219946 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0308f9-9426-4d00-b95d-a9a7e6c8dc4b None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:20.220168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0308f9-9426-4d00-b95d-a9a7e6c8dc4b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:20.231093 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6fd135-1d2a-4ed3-96d7-48d4f69ae15d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:20.232233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0308f9-9426-4d00-b95d-a9a7e6c8dc4b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:20.235970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-9b6fd135-1d2a-4ed3-96d7-48d4f69ae15d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 30 00:43:20.235970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:20.235970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:20.235970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:20.235970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:20.235970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:20.235970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:20.235970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:20.235970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:20.235970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:20.235970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:20.235970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:20.235970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:20.235970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:20.235970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:20.235970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:20.236832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:20.236832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:20.236832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:20.236832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:20.236832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:20.236832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:20.236832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:20.236832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:20.236832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:20.236832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:20.236832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:20.236832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:20.236832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:20.236832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:20.236832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:20.236832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:20.237681 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:20.237681 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:20.237681 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:20.237681 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:20.237681 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:20.237681 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:20.237681 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:20.237681 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:20.237681 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:20.237681 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:20.237681 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 30 00:43:20.237681 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:20.237681 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 157/311] 10.222.0.16 () {56 vars in 1114 bytes} [Sun Oct 30 00:43:20 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:20.238320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-2c0308f9-9426-4d00-b95d-a9a7e6c8dc4b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:43:20.238320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:20.238320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:20.238320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:20.238320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:20.238320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:20.238320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:20.238320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:20.238320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:20.238320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:20.238320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:20.238320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:20.238320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:20.238320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:20.238320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:20.238320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:20.239805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:20.239805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:20.239805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:20.239805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:20.239805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:20.239805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:20.239805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:20.239805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:20.239805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:20.239805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:20.239805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:20.239805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:20.239805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:20.239805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:20.239805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:20.239805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:20.240616 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:20.240616 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:20.240616 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:20.240616 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:20.240616 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:20.240616 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:20.240616 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:20.240616 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:20.240616 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:20.240616 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:20.240616 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 30 00:43:20.240616 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:20.241198 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6fd135-1d2a-4ed3-96d7-48d4f69ae15d None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:20.241267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 155/312] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:43:20 2022] GET /identity/v3/roles/member => generated 84 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:20.244214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0308f9-9426-4d00-b95d-a9a7e6c8dc4b None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:20.256103 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:20.358049 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6fd135-1d2a-4ed3-96d7-48d4f69ae15d None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:20.367908 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0308f9-9426-4d00-b95d-a9a7e6c8dc4b None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:20.375644 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:20.401534 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fb69e0-2b10-45c3-aa49-4e5a1ba56dfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:20.402696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fb69e0-2b10-45c3-aa49-4e5a1ba56dfb None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:20.402971 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fb69e0-2b10-45c3-aa49-4e5a1ba56dfb None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:20.403242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fb69e0-2b10-45c3-aa49-4e5a1ba56dfb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:20.407296 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9c41b1-7b04-45d5-87da-b7f5004ce4ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:20.407981 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9c41b1-7b04-45d5-87da-b7f5004ce4ff None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:20.408254 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9c41b1-7b04-45d5-87da-b7f5004ce4ff None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:20.408523 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9c41b1-7b04-45d5-87da-b7f5004ce4ff None admin] PATH_INFO: `/v3/groups/23e7e0b4e22f4a35ac36c35c239812fd` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:20.409694 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9c41b1-7b04-45d5-87da-b7f5004ce4ff None admin] RBAC: Authorizing `identity:get_group(group_id=23e7e0b4e22f4a35ac36c35c239812fd)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:20.411135 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9c41b1-7b04-45d5-87da-b7f5004ce4ff None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:20.412899 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 156/313] 10.222.0.16 () {56 vars in 1180 bytes} [Sun Oct 30 00:43:20 2022] GET /identity/v3/groups/23e7e0b4e22f4a35ac36c35c239812fd => generated 226 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:20.413474 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9c41b1-7b04-45d5-87da-b7f5004ce4ff None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:20.478045 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07fb69e0-2b10-45c3-aa49-4e5a1ba56dfb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:20.479820 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07fb69e0-2b10-45c3-aa49-4e5a1ba56dfb None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:20.502257 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:20.536008 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9c41b1-7b04-45d5-87da-b7f5004ce4ff None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:20.575160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d3b262-fd00-4a8d-a2e4-dd2e54c46b80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:20.576277 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d3b262-fd00-4a8d-a2e4-dd2e54c46b80 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:20.576692 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d3b262-fd00-4a8d-a2e4-dd2e54c46b80 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:20.577092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d3b262-fd00-4a8d-a2e4-dd2e54c46b80 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:20.577935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d3b262-fd00-4a8d-a2e4-dd2e54c46b80 None admin] RBAC: Adding query filter params (name=member) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:20.578104 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d3b262-fd00-4a8d-a2e4-dd2e54c46b80 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:20.579474 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d3b262-fd00-4a8d-a2e4-dd2e54c46b80 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:20.588517 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 157/315] 10.222.0.16 () {56 vars in 1101 bytes} [Sun Oct 30 00:43:20 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:20.588677 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 158/315] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:43:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 30 00:43:20.591507 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fb69e0-2b10-45c3-aa49-4e5a1ba56dfb None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:20.591621 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92da376d-88b7-4fed-b8b7-dbf16067268e None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:20.591621 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92da376d-88b7-4fed-b8b7-dbf16067268e None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:20.591621 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92da376d-88b7-4fed-b8b7-dbf16067268e None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:20.592741 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 158/316] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:20.593265 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92da376d-88b7-4fed-b8b7-dbf16067268e None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:20.612474 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:20.707676 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fb69e0-2b10-45c3-aa49-4e5a1ba56dfb None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:20.717252 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92da376d-88b7-4fed-b8b7-dbf16067268e None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:20.745020 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8b05d3-c78b-4488-8649-5f03c92b5a22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:20.745627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8b05d3-c78b-4488-8649-5f03c92b5a22 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:20.745905 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8b05d3-c78b-4488-8649-5f03c92b5a22 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:20.746171 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8b05d3-c78b-4488-8649-5f03c92b5a22 None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:20.746828 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8b05d3-c78b-4488-8649-5f03c92b5a22 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:20.747120 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8b05d3-c78b-4488-8649-5f03c92b5a22 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:20.748540 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8b05d3-c78b-4488-8649-5f03c92b5a22 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:20.757115 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 159/317] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:43:20 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:20.758007 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8b05d3-c78b-4488-8649-5f03c92b5a22 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:20.758219 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1e02a8-f420-474a-b622-d8921eb90234 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:20.758888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1e02a8-f420-474a-b622-d8921eb90234 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:20.759166 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1e02a8-f420-474a-b622-d8921eb90234 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:20.759433 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1e02a8-f420-474a-b622-d8921eb90234 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:20.760187 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1e02a8-f420-474a-b622-d8921eb90234 None admin] RBAC: Adding query filter params (group.id=23e7e0b4e22f4a35ac36c35c239812fd, role.id=5c19ef44708c4d1f95f5f657c5c79361, scope.project.id=e48719636cc1460283c3bc75e46b4a17) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:20.760492 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1e02a8-f420-474a-b622-d8921eb90234 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:20.761933 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1e02a8-f420-474a-b622-d8921eb90234 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:20.765041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:20.775751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 159/318] 10.222.0.16 () {56 vars in 1376 bytes} [Sun Oct 30 00:43:20 2022] GET /identity/v3/role_assignments?group.id=23e7e0b4e22f4a35ac36c35c239812fd&scope.project.id=e48719636cc1460283c3bc75e46b4a17&role.id=5c19ef44708c4d1f95f5f657c5c79361 => generated 611 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:20.775751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efa6b6a-c46f-4eb2-bb98-62bf02e6f33e None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:20.775751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efa6b6a-c46f-4eb2-bb98-62bf02e6f33e None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:20.775751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efa6b6a-c46f-4eb2-bb98-62bf02e6f33e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:20.796291 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-4efa6b6a-c46f-4eb2-bb98-62bf02e6f33e None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:20.797565 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-4efa6b6a-c46f-4eb2-bb98-62bf02e6f33e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:20.862663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:20.885096 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 160/319] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:20.885966 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efa6b6a-c46f-4eb2-bb98-62bf02e6f33e None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:20.899191 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8b05d3-c78b-4488-8649-5f03c92b5a22 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:20.948748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9575c869-caf0-4a71-a1fa-45c987e4ed74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:20.949845 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9575c869-caf0-4a71-a1fa-45c987e4ed74 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:20.950677 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9575c869-caf0-4a71-a1fa-45c987e4ed74 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:20.951787 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9575c869-caf0-4a71-a1fa-45c987e4ed74 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:20.998875 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efa6b6a-c46f-4eb2-bb98-62bf02e6f33e None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:21.032649 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9575c869-caf0-4a71-a1fa-45c987e4ed74 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:21.034364 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9575c869-caf0-4a71-a1fa-45c987e4ed74 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:21.039261 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:21.050111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b357acb-0757-4ddc-8017-cece522cb377 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:21.051139 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b357acb-0757-4ddc-8017-cece522cb377 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:21.051684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b357acb-0757-4ddc-8017-cece522cb377 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:21.052047 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b357acb-0757-4ddc-8017-cece522cb377 None admin] PATH_INFO: `/v3/projects/e48719636cc1460283c3bc75e46b4a17` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:21.053168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b357acb-0757-4ddc-8017-cece522cb377 None admin] RBAC: Authorizing `identity:get_project(project_id=e48719636cc1460283c3bc75e46b4a17)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:21.054587 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b357acb-0757-4ddc-8017-cece522cb377 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:21.063400 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 161/320] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:43:20 2022] GET /identity/v3/projects/e48719636cc1460283c3bc75e46b4a17 => generated 298 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:21.063400 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b357acb-0757-4ddc-8017-cece522cb377 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:21.130416 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:21.167961 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 160/321] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:43:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 30 00:43:21.168687 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9575c869-caf0-4a71-a1fa-45c987e4ed74 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:21.187278 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b357acb-0757-4ddc-8017-cece522cb377 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:21.225466 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc6d88f-f6e4-416b-b295-bbdc43c01657 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:21.226376 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc6d88f-f6e4-416b-b295-bbdc43c01657 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:21.226835 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc6d88f-f6e4-416b-b295-bbdc43c01657 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:21.227101 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc6d88f-f6e4-416b-b295-bbdc43c01657 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:21.281064 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9575c869-caf0-4a71-a1fa-45c987e4ed74 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:21.291954 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:21.295415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc6d88f-f6e4-416b-b295-bbdc43c01657 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:21.297283 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc6d88f-f6e4-416b-b295-bbdc43c01657 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:21.315618 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80914cac-ed4c-43f0-9ef9-630132bb6088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:21.316270 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80914cac-ed4c-43f0-9ef9-630132bb6088 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:21.316548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80914cac-ed4c-43f0-9ef9-630132bb6088 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:21.316817 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80914cac-ed4c-43f0-9ef9-630132bb6088 None admin] PATH_INFO: `/v3/groups/23e7e0b4e22f4a35ac36c35c239812fd` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:21.317924 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80914cac-ed4c-43f0-9ef9-630132bb6088 None admin] RBAC: Authorizing `identity:get_group(group_id=23e7e0b4e22f4a35ac36c35c239812fd)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:21.319362 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80914cac-ed4c-43f0-9ef9-630132bb6088 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:21.321150 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 161/322] 10.222.0.16 () {56 vars in 1180 bytes} [Sun Oct 30 00:43:21 2022] GET /identity/v3/groups/23e7e0b4e22f4a35ac36c35c239812fd => generated 226 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:21.321818 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80914cac-ed4c-43f0-9ef9-630132bb6088 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:21.363612 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:21.404478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 162/323] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:43:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 30 00:43:21.405219 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc6d88f-f6e4-416b-b295-bbdc43c01657 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:21.425097 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80914cac-ed4c-43f0-9ef9-630132bb6088 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:21.458729 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170fe057-fc0e-4732-ad61-b08497ba8d6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:21.459345 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170fe057-fc0e-4732-ad61-b08497ba8d6e None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:21.459613 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170fe057-fc0e-4732-ad61-b08497ba8d6e None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:21.459903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170fe057-fc0e-4732-ad61-b08497ba8d6e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:21.470143 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-170fe057-fc0e-4732-ad61-b08497ba8d6e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:21.475021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-170fe057-fc0e-4732-ad61-b08497ba8d6e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:43:21.475021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:21.475021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:21.475021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:21.475021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:21.475021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:21.475021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:21.475021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:21.475021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:21.475021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:21.475021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:21.475021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:21.475021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:21.475021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:21.475021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:21.475021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:21.476015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:21.476015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:21.476015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:21.476015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:21.476015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:21.476015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:21.476015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:21.476015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:21.476015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:21.476015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:21.476015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:21.476015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:21.476015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:21.476015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:21.476015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:21.476015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:21.476811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:21.476811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:21.476811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:21.476811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:21.476811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:21.476811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:21.476811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:21.476811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:21.476811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:21.476811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:21.476811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:43:21.476811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:21.476811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 162/324] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:43:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:21.478695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170fe057-fc0e-4732-ad61-b08497ba8d6e None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:21.497245 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:21.501275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc6d88f-f6e4-416b-b295-bbdc43c01657 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:21.539670 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46afce36-72ca-4d7b-99c3-0d99af040987 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:21.540259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46afce36-72ca-4d7b-99c3-0d99af040987 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:21.540533 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46afce36-72ca-4d7b-99c3-0d99af040987 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:21.540800 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46afce36-72ca-4d7b-99c3-0d99af040987 None admin] PATH_INFO: `/v3/projects/ab2a3d04642247e2895bcc3311a9a205` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:21.541926 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46afce36-72ca-4d7b-99c3-0d99af040987 None admin] RBAC: Authorizing `identity:get_project(project_id=ab2a3d04642247e2895bcc3311a9a205)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:21.543373 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46afce36-72ca-4d7b-99c3-0d99af040987 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:21.545025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 163/325] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:43:21 2022] GET /identity/v3/projects/ab2a3d04642247e2895bcc3311a9a205 => generated 302 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:21.550576 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46afce36-72ca-4d7b-99c3-0d99af040987 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:21.581320 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170fe057-fc0e-4732-ad61-b08497ba8d6e None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:21.582808 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:21.615614 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e67d98-3d74-429d-8b94-46ce5ef9ee3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:21.616263 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e67d98-3d74-429d-8b94-46ce5ef9ee3d None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:21.616535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e67d98-3d74-429d-8b94-46ce5ef9ee3d None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:21.616792 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e67d98-3d74-429d-8b94-46ce5ef9ee3d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:21.617522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e67d98-3d74-429d-8b94-46ce5ef9ee3d None admin] RBAC: Adding query filter params (group.id=23e7e0b4e22f4a35ac36c35c239812fd, role.id=22e839a60f6f414abd42c577025184a2, scope.project.id=e48719636cc1460283c3bc75e46b4a17) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:21.617822 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e67d98-3d74-429d-8b94-46ce5ef9ee3d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:21.619386 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e67d98-3d74-429d-8b94-46ce5ef9ee3d None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:21.639230 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 163/326] 10.222.0.16 () {56 vars in 1376 bytes} [Sun Oct 30 00:43:21 2022] GET /identity/v3/role_assignments?group.id=23e7e0b4e22f4a35ac36c35c239812fd&scope.project.id=e48719636cc1460283c3bc75e46b4a17&role.id=22e839a60f6f414abd42c577025184a2 => generated 611 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:21.639230 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e67d98-3d74-429d-8b94-46ce5ef9ee3d None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:21.652346 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46afce36-72ca-4d7b-99c3-0d99af040987 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:21.693436 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee77160e-8734-4283-98c4-f1b36326b7e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:21.694081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee77160e-8734-4283-98c4-f1b36326b7e8 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:21.694352 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee77160e-8734-4283-98c4-f1b36326b7e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:21.694616 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee77160e-8734-4283-98c4-f1b36326b7e8 None admin] PATH_INFO: `/v3/projects/ab2a3d04642247e2895bcc3311a9a205` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:21.695753 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee77160e-8734-4283-98c4-f1b36326b7e8 None admin] RBAC: Authorizing `identity:get_project(project_id=ab2a3d04642247e2895bcc3311a9a205)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:21.697194 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee77160e-8734-4283-98c4-f1b36326b7e8 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:21.703937 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 164/327] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:43:21 2022] GET /identity/v3/projects/ab2a3d04642247e2895bcc3311a9a205 => generated 302 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:21.703937 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee77160e-8734-4283-98c4-f1b36326b7e8 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:21.744084 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:21.747656 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e67d98-3d74-429d-8b94-46ce5ef9ee3d None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:21.785099 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31dd5bf4-0e8c-4b73-b10d-507aee3608ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:21.785721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31dd5bf4-0e8c-4b73-b10d-507aee3608ab None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:21.785998 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31dd5bf4-0e8c-4b73-b10d-507aee3608ab None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:21.786268 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31dd5bf4-0e8c-4b73-b10d-507aee3608ab None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:21.786939 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31dd5bf4-0e8c-4b73-b10d-507aee3608ab None admin] RBAC: Adding query filter params (name=admin) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:21.787251 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31dd5bf4-0e8c-4b73-b10d-507aee3608ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:21.788696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31dd5bf4-0e8c-4b73-b10d-507aee3608ab None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:21.804632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 164/328] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:43:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:21.804632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31dd5bf4-0e8c-4b73-b10d-507aee3608ab None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:21.837925 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee77160e-8734-4283-98c4-f1b36326b7e8 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:21.843837 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:21.883914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0384517-a99c-43fe-9492-17f71dececc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:21.885059 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0384517-a99c-43fe-9492-17f71dececc7 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:21.885391 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0384517-a99c-43fe-9492-17f71dececc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:21.885676 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0384517-a99c-43fe-9492-17f71dececc7 None admin] PATH_INFO: `/v3/groups/23e7e0b4e22f4a35ac36c35c239812fd` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:21.887274 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0384517-a99c-43fe-9492-17f71dececc7 None admin] RBAC: Authorizing `identity:get_group(group_id=23e7e0b4e22f4a35ac36c35c239812fd)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:21.890674 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0384517-a99c-43fe-9492-17f71dececc7 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:21.890674 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 165/329] 10.222.0.16 () {56 vars in 1180 bytes} [Sun Oct 30 00:43:21 2022] GET /identity/v3/groups/23e7e0b4e22f4a35ac36c35c239812fd => generated 226 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:21.891378 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0384517-a99c-43fe-9492-17f71dececc7 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:21.946907 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31dd5bf4-0e8c-4b73-b10d-507aee3608ab None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:21.993246 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5490dd3-e886-47a5-b473-3dc38de658de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:21.994015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5490dd3-e886-47a5-b473-3dc38de658de None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:21.994588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5490dd3-e886-47a5-b473-3dc38de658de None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:21.994588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5490dd3-e886-47a5-b473-3dc38de658de None admin] PATH_INFO: `/v3/groups/23e7e0b4e22f4a35ac36c35c239812fd` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:21.996090 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5490dd3-e886-47a5-b473-3dc38de658de None admin] RBAC: Authorizing `identity:get_group(group_id=23e7e0b4e22f4a35ac36c35c239812fd)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:21.997307 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5490dd3-e886-47a5-b473-3dc38de658de None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:21.999054 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 165/330] 10.222.0.16 () {56 vars in 1180 bytes} [Sun Oct 30 00:43:21 2022] GET /identity/v3/groups/23e7e0b4e22f4a35ac36c35c239812fd => generated 226 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:22.000500 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5490dd3-e886-47a5-b473-3dc38de658de None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:22.031236 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0384517-a99c-43fe-9492-17f71dececc7 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:22.033253 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:22.069712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a3900d-5e99-42fc-b37b-ce820ef847dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:22.070231 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a3900d-5e99-42fc-b37b-ce820ef847dd None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:22.070501 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a3900d-5e99-42fc-b37b-ce820ef847dd None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:22.070761 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a3900d-5e99-42fc-b37b-ce820ef847dd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:22.106614 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:22.113096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5490dd3-e886-47a5-b473-3dc38de658de None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:22.139066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a3900d-5e99-42fc-b37b-ce820ef847dd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:22.140550 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a3900d-5e99-42fc-b37b-ce820ef847dd None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:22.150606 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139fac21-6265-410e-bfd8-51e890cbfcd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:22.151205 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139fac21-6265-410e-bfd8-51e890cbfcd6 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:22.151482 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139fac21-6265-410e-bfd8-51e890cbfcd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:22.151739 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139fac21-6265-410e-bfd8-51e890cbfcd6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:22.152487 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139fac21-6265-410e-bfd8-51e890cbfcd6 None admin] RBAC: Adding query filter params (group.id=23e7e0b4e22f4a35ac36c35c239812fd, role.id=5c19ef44708c4d1f95f5f657c5c79361, scope.project.id=ab2a3d04642247e2895bcc3311a9a205) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:22.152785 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139fac21-6265-410e-bfd8-51e890cbfcd6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:22.154197 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139fac21-6265-410e-bfd8-51e890cbfcd6 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:22.160823 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 166/331] 10.222.0.16 () {56 vars in 1376 bytes} [Sun Oct 30 00:43:21 2022] GET /identity/v3/role_assignments?group.id=23e7e0b4e22f4a35ac36c35c239812fd&scope.project.id=ab2a3d04642247e2895bcc3311a9a205&role.id=5c19ef44708c4d1f95f5f657c5c79361 => generated 611 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:22.161740 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139fac21-6265-410e-bfd8-51e890cbfcd6 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:22.267778 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 166/332] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:43:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 30 00:43:22.274752 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a3900d-5e99-42fc-b37b-ce820ef847dd None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:22.277892 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:22.301765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139fac21-6265-410e-bfd8-51e890cbfcd6 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:22.350354 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87b163d-8e53-4aaf-82c9-3a1a5acb8ba7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:22.350954 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87b163d-8e53-4aaf-82c9-3a1a5acb8ba7 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:22.351269 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87b163d-8e53-4aaf-82c9-3a1a5acb8ba7 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:22.351494 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87b163d-8e53-4aaf-82c9-3a1a5acb8ba7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:22.354420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87b163d-8e53-4aaf-82c9-3a1a5acb8ba7 None admin] RBAC: Adding query filter params (group.id=23e7e0b4e22f4a35ac36c35c239812fd, role.id=22e839a60f6f414abd42c577025184a2, scope.project.id=ab2a3d04642247e2895bcc3311a9a205) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:22.354420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87b163d-8e53-4aaf-82c9-3a1a5acb8ba7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:22.354420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87b163d-8e53-4aaf-82c9-3a1a5acb8ba7 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:22.361032 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 167/333] 10.222.0.16 () {56 vars in 1376 bytes} [Sun Oct 30 00:43:22 2022] GET /identity/v3/role_assignments?group.id=23e7e0b4e22f4a35ac36c35c239812fd&scope.project.id=ab2a3d04642247e2895bcc3311a9a205&role.id=22e839a60f6f414abd42c577025184a2 => generated 611 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:22.413577 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a3900d-5e99-42fc-b37b-ce820ef847dd None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:22.465896 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbda008-9fc7-4774-a981-b46123a05d67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:22.466556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbda008-9fc7-4774-a981-b46123a05d67 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:22.466831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbda008-9fc7-4774-a981-b46123a05d67 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:22.467099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbda008-9fc7-4774-a981-b46123a05d67 None admin] PATH_INFO: `/v3/projects/3907972f9f6248debeeda8ecd254820e` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:22.468958 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bbda008-9fc7-4774-a981-b46123a05d67 None admin] RBAC: Authorizing `identity:get_project(project_id=3907972f9f6248debeeda8ecd254820e)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:22.469915 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bbda008-9fc7-4774-a981-b46123a05d67 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:22.472698 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 167/334] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:43:22 2022] GET /identity/v3/projects/3907972f9f6248debeeda8ecd254820e => generated 344 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:22.478498 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87b163d-8e53-4aaf-82c9-3a1a5acb8ba7 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:22.510921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:22.593151 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87b163d-8e53-4aaf-82c9-3a1a5acb8ba7 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:22.626357 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0daaca-ddd7-493f-80b9-bd4e03fb8db6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:22.626921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0daaca-ddd7-493f-80b9-bd4e03fb8db6 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:22.627192 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0daaca-ddd7-493f-80b9-bd4e03fb8db6 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:22.627459 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0daaca-ddd7-493f-80b9-bd4e03fb8db6 None admin] PATH_INFO: `/v3/groups/0cb476b7bf89498e95bd5a2dfae11222` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:22.628687 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0daaca-ddd7-493f-80b9-bd4e03fb8db6 None admin] RBAC: Authorizing `identity:get_group(group_id=0cb476b7bf89498e95bd5a2dfae11222)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:22.630118 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0daaca-ddd7-493f-80b9-bd4e03fb8db6 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:22.632097 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 168/335] 10.222.0.16 () {56 vars in 1180 bytes} [Sun Oct 30 00:43:22 2022] GET /identity/v3/groups/0cb476b7bf89498e95bd5a2dfae11222 => generated 229 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:22.637682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbda008-9fc7-4774-a981-b46123a05d67 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:22.709622 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:22.741167 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbda008-9fc7-4774-a981-b46123a05d67 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:22.777619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71af3fe-ff19-4c21-821e-b4cfa1c6d3dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:22.778221 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71af3fe-ff19-4c21-821e-b4cfa1c6d3dd None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:22.778494 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71af3fe-ff19-4c21-821e-b4cfa1c6d3dd None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:22.778758 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71af3fe-ff19-4c21-821e-b4cfa1c6d3dd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:22.779477 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71af3fe-ff19-4c21-821e-b4cfa1c6d3dd None admin] RBAC: Adding query filter params (group.id=0cb476b7bf89498e95bd5a2dfae11222, role.id=35f59686b5d84291a8d9baf19287eb5a, scope.project.id=3907972f9f6248debeeda8ecd254820e) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:22.779862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71af3fe-ff19-4c21-821e-b4cfa1c6d3dd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:22.781390 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71af3fe-ff19-4c21-821e-b4cfa1c6d3dd None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:22.787504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 168/336] 10.222.0.16 () {56 vars in 1376 bytes} [Sun Oct 30 00:43:22 2022] GET /identity/v3/role_assignments?group.id=0cb476b7bf89498e95bd5a2dfae11222&scope.project.id=3907972f9f6248debeeda8ecd254820e&role.id=35f59686b5d84291a8d9baf19287eb5a => generated 611 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:25.295219 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c0e89a-fc36-4381-a479-0f47bb44e546 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:25.296254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c0e89a-fc36-4381-a479-0f47bb44e546 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:25.296748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c0e89a-fc36-4381-a479-0f47bb44e546 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:25.298599 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 169/337] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:25.304539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a19be01-2418-4416-82dc-f6b484c48521 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:25.305003 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a19be01-2418-4416-82dc-f6b484c48521 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:25.305078 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a19be01-2418-4416-82dc-f6b484c48521 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:25.327913 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-2a19be01-2418-4416-82dc-f6b484c48521 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:25.329108 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-2a19be01-2418-4416-82dc-f6b484c48521 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:25.419268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 169/338] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:25.596070 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6889535c-c5b3-4063-b643-1e33c91d286b None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:25.596307 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6889535c-c5b3-4063-b643-1e33c91d286b None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:25.596612 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6889535c-c5b3-4063-b643-1e33c91d286b None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:25.598004 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 170/339] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:25.603788 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74a04fa-821a-42fd-91ca-c2254f6b306e None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:25.604374 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74a04fa-821a-42fd-91ca-c2254f6b306e None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:25.604835 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74a04fa-821a-42fd-91ca-c2254f6b306e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:25.632995 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-a74a04fa-821a-42fd-91ca-c2254f6b306e None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:25.634585 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-a74a04fa-821a-42fd-91ca-c2254f6b306e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:25.708318 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:25.717547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 170/340] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:25.754268 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957cdf80-0bea-4a13-98fb-e1b20fda17b0 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:25.754584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957cdf80-0bea-4a13-98fb-e1b20fda17b0 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:25.754810 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957cdf80-0bea-4a13-98fb-e1b20fda17b0 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:25.756459 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 171/341] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:25.761801 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d24bb1-abfe-417e-a321-25ce2371851f None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:25.762081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d24bb1-abfe-417e-a321-25ce2371851f None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:25.762350 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d24bb1-abfe-417e-a321-25ce2371851f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:25.780472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-94d24bb1-abfe-417e-a321-25ce2371851f None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:25.781646 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-94d24bb1-abfe-417e-a321-25ce2371851f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:25.841081 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957cdf80-0bea-4a13-98fb-e1b20fda17b0 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:25.874628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 171/342] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:43:25.951320 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:25.964676 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957cdf80-0bea-4a13-98fb-e1b20fda17b0 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:26.011324 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8e2c66-195b-4706-a3a3-5f6ab618df40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:26.012014 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8e2c66-195b-4706-a3a3-5f6ab618df40 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:26.012335 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8e2c66-195b-4706-a3a3-5f6ab618df40 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:26.012601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8e2c66-195b-4706-a3a3-5f6ab618df40 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:26.025167 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8e2c66-195b-4706-a3a3-5f6ab618df40 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:26.026618 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8e2c66-195b-4706-a3a3-5f6ab618df40 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:26.046984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-fa8e2c66-195b-4706-a3a3-5f6ab618df40 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:26.046984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:26.046984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:26.046984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:26.046984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:26.046984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:26.046984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:26.046984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:26.046984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:26.046984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:26.046984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:26.046984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:26.046984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:26.046984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:26.046984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:26.046984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:26.047992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:26.047992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:26.047992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:26.047992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:26.047992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:26.047992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:26.047992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:26.047992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:26.047992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:26.047992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:26.047992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:26.047992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:26.047992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:26.047992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:26.047992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:26.047992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:26.047992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:26.048865 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:26.048865 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:26.048865 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:26.048865 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 172/343] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:26.055517 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d24bb1-abfe-417e-a321-25ce2371851f None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:26.130409 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8e2c66-195b-4706-a3a3-5f6ab618df40 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:26.209274 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d24bb1-abfe-417e-a321-25ce2371851f None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:26.244068 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:26.264083 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87dd1ed-481f-46c1-a029-0cc6c4df4c28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:26.264692 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87dd1ed-481f-46c1-a029-0cc6c4df4c28 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:26.264962 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87dd1ed-481f-46c1-a029-0cc6c4df4c28 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:26.265232 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87dd1ed-481f-46c1-a029-0cc6c4df4c28 None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:26.265894 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87dd1ed-481f-46c1-a029-0cc6c4df4c28 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:26.266185 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87dd1ed-481f-46c1-a029-0cc6c4df4c28 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:26.267560 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87dd1ed-481f-46c1-a029-0cc6c4df4c28 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:26.288869 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 172/344] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:26.289699 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87dd1ed-481f-46c1-a029-0cc6c4df4c28 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:26.330783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8e2c66-195b-4706-a3a3-5f6ab618df40 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:26.379081 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97396e1-f923-4411-88c0-0c4845a54ee4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:26.379732 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97396e1-f923-4411-88c0-0c4845a54ee4 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:26.380032 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97396e1-f923-4411-88c0-0c4845a54ee4 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:26.380303 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97396e1-f923-4411-88c0-0c4845a54ee4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:26.390979 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97396e1-f923-4411-88c0-0c4845a54ee4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:26.392384 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97396e1-f923-4411-88c0-0c4845a54ee4 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:26.394908 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:26.403327 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87dd1ed-481f-46c1-a029-0cc6c4df4c28 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:26.407936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-a97396e1-f923-4411-88c0-0c4845a54ee4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:26.407936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:26.407936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:26.407936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:26.407936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:26.407936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:26.407936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:26.407936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:26.407936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:26.407936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:26.407936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:26.407936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:26.407936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:26.407936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:26.407936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:26.407936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:26.409043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:26.409043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:26.409043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:26.409043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:26.409043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:26.409043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:26.409043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:26.409043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:26.409043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:26.409043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:26.409043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:26.409043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:26.409043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:26.409043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:26.409043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:26.409043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:26.409043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:26.409941 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:26.409941 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:26.409941 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:26.409941 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 173/345] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:26.410162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97396e1-f923-4411-88c0-0c4845a54ee4 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:26.441390 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bacc879-ad40-4ec2-8d30-e3fe1489e520 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:26.442018 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bacc879-ad40-4ec2-8d30-e3fe1489e520 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:26.442290 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bacc879-ad40-4ec2-8d30-e3fe1489e520 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:26.442556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bacc879-ad40-4ec2-8d30-e3fe1489e520 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:26.454502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bacc879-ad40-4ec2-8d30-e3fe1489e520 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:26.456017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bacc879-ad40-4ec2-8d30-e3fe1489e520 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:26.470230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-3bacc879-ad40-4ec2-8d30-e3fe1489e520 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:26.470230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:26.470230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:26.470230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:26.470230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:26.470230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:26.470230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:26.470230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:26.470230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:26.470230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:26.470230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:26.470230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:26.470230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:26.470230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:26.470230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:26.470230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:26.471193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:26.471193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:26.471193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:26.471193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:26.471193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:26.471193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:26.471193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:26.471193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:26.471193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:26.471193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:26.471193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:26.471193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:26.471193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:26.471193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:26.471193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:26.471193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:26.471193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:26.472447 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:26.472447 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:26.472447 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:26.472447 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 173/346] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:26.472682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784a1a20-cbcf-4385-9d35-43b6d67e6808 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:26.472753 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784a1a20-cbcf-4385-9d35-43b6d67e6808 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:26.473118 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784a1a20-cbcf-4385-9d35-43b6d67e6808 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:26.474359 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 174/347] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:26 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:26.478885 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784a1a20-cbcf-4385-9d35-43b6d67e6808 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:26.520909 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97396e1-f923-4411-88c0-0c4845a54ee4 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:26.522535 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:26.554867 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da09155c-e049-4d62-ad37-7adf96bcfcd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:26.555480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da09155c-e049-4d62-ad37-7adf96bcfcd2 None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:26.555772 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da09155c-e049-4d62-ad37-7adf96bcfcd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:26.556062 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da09155c-e049-4d62-ad37-7adf96bcfcd2 None admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:26.556689 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da09155c-e049-4d62-ad37-7adf96bcfcd2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:26.558203 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da09155c-e049-4d62-ad37-7adf96bcfcd2 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:26.581793 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 174/348] 10.222.0.16 () {60 vars in 1132 bytes} [Sun Oct 30 00:43:26 2022] POST /identity/v3/users => generated 246 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:43:26.583010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da09155c-e049-4d62-ad37-7adf96bcfcd2 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:26.607014 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784a1a20-cbcf-4385-9d35-43b6d67e6808 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:26.647785 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:26.661769 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ced26bb-f4ce-4524-bbf1-160854975990 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:26.662609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ced26bb-f4ce-4524-bbf1-160854975990 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:26.662948 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ced26bb-f4ce-4524-bbf1-160854975990 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:26.663276 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ced26bb-f4ce-4524-bbf1-160854975990 None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:26.664030 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ced26bb-f4ce-4524-bbf1-160854975990 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:26.664378 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ced26bb-f4ce-4524-bbf1-160854975990 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:26.665851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ced26bb-f4ce-4524-bbf1-160854975990 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:26.681333 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 175/349] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:26.682904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cfdb73-d77d-4093-9268-42a3864db586 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:26.683251 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cfdb73-d77d-4093-9268-42a3864db586 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:26.683583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cfdb73-d77d-4093-9268-42a3864db586 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:26.706155 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-c0cfdb73-d77d-4093-9268-42a3864db586 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:26.707881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c0cfdb73-d77d-4093-9268-42a3864db586 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:26.712946 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da09155c-e049-4d62-ad37-7adf96bcfcd2 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:26.760652 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13484f15-5149-4233-88e9-57f723871090 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:26.761598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13484f15-5149-4233-88e9-57f723871090 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:26.762566 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13484f15-5149-4233-88e9-57f723871090 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:26.763033 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13484f15-5149-4233-88e9-57f723871090 None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:26.764105 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13484f15-5149-4233-88e9-57f723871090 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:26.764436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13484f15-5149-4233-88e9-57f723871090 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:26.765853 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13484f15-5149-4233-88e9-57f723871090 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:26.783177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 175/350] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:26.785178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13484f15-5149-4233-88e9-57f723871090 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:26.804415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 176/351] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:26.805302 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cfdb73-d77d-4093-9268-42a3864db586 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:26.840649 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:26.901070 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13484f15-5149-4233-88e9-57f723871090 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:26.916597 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:26.940373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f693e598-b99f-4b64-a1f0-511db335693f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:26.941291 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f693e598-b99f-4b64-a1f0-511db335693f None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:26.941564 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f693e598-b99f-4b64-a1f0-511db335693f None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:26.941830 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f693e598-b99f-4b64-a1f0-511db335693f None admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:26.942742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f693e598-b99f-4b64-a1f0-511db335693f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:26.944172 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cfdb73-d77d-4093-9268-42a3864db586 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:26.944693 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f693e598-b99f-4b64-a1f0-511db335693f None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:26.973992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 176/352] 10.222.0.16 () {60 vars in 1132 bytes} [Sun Oct 30 00:43:26 2022] POST /identity/v3/users => generated 248 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:43:26.981079 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4789cbd7-f931-447d-8658-6e35ce099134 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:26.981365 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4789cbd7-f931-447d-8658-6e35ce099134 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:26.981799 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4789cbd7-f931-447d-8658-6e35ce099134 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:26.984259 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 177/353] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:26 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:26.990402 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082e9dab-21ca-4f4b-99c9-420a8567251f None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:26.990826 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082e9dab-21ca-4f4b-99c9-420a8567251f None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:26.991102 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082e9dab-21ca-4f4b-99c9-420a8567251f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:27.011525 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-082e9dab-21ca-4f4b-99c9-420a8567251f None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:27.012524 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f7ede7-bf81-4626-b971-d2614c1963b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:27.012811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-082e9dab-21ca-4f4b-99c9-420a8567251f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:27.013157 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f7ede7-bf81-4626-b971-d2614c1963b3 None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:27.013431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f7ede7-bf81-4626-b971-d2614c1963b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:27.013701 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f7ede7-bf81-4626-b971-d2614c1963b3 None admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:27.014359 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f7ede7-bf81-4626-b971-d2614c1963b3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:27.015829 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f7ede7-bf81-4626-b971-d2614c1963b3 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:27.066535 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 177/354] 10.222.0.16 () {60 vars in 1132 bytes} [Sun Oct 30 00:43:26 2022] POST /identity/v3/users => generated 248 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:43:27.110974 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 178/355] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:43:27.127300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f7ede7-bf81-4626-b971-d2614c1963b3 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:27.222158 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:27.243653 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f7ede7-bf81-4626-b971-d2614c1963b3 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:27.283816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb2c9fc-561b-48a0-ab85-5146f18e5f35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:27.284832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb2c9fc-561b-48a0-ab85-5146f18e5f35 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:27.285293 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb2c9fc-561b-48a0-ab85-5146f18e5f35 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:27.285739 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb2c9fc-561b-48a0-ab85-5146f18e5f35 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:27.297474 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb2c9fc-561b-48a0-ab85-5146f18e5f35 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:27.299375 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb2c9fc-561b-48a0-ab85-5146f18e5f35 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:27.315213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-bfb2c9fc-561b-48a0-ab85-5146f18e5f35 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:27.315213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:27.315213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:27.315213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:27.315213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:27.315213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:27.315213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:27.315213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:27.315213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:27.315213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:27.315213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:27.315213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:27.315213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:27.315213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:27.315213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:27.315213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:27.316339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:27.316339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:27.316339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:27.316339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:27.316339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:27.316339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:27.316339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:27.316339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:27.316339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:27.316339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:27.316339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:27.316339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:27.316339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:27.316339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:27.316339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:27.316339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:27.316339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:27.317182 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:27.317182 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:27.317182 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:27.318434 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 178/356] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:27.324798 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082e9dab-21ca-4f4b-99c9-420a8567251f None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:27.428695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082e9dab-21ca-4f4b-99c9-420a8567251f None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:27.431082 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:27.470474 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb2c9fc-561b-48a0-ab85-5146f18e5f35 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:27.475744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c738a6-645a-46e3-b4aa-9de66b69068a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:27.476336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c738a6-645a-46e3-b4aa-9de66b69068a None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:27.476606 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c738a6-645a-46e3-b4aa-9de66b69068a None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:27.476870 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c738a6-645a-46e3-b4aa-9de66b69068a None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:27.477514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c738a6-645a-46e3-b4aa-9de66b69068a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:27.477852 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c738a6-645a-46e3-b4aa-9de66b69068a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:27.479493 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c738a6-645a-46e3-b4aa-9de66b69068a None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:27.500422 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 179/357] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:27.505461 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c738a6-645a-46e3-b4aa-9de66b69068a None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:27.596317 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb2c9fc-561b-48a0-ab85-5146f18e5f35 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:27.631679 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309b4c3b-b659-4a9e-a56d-7e91513aa519 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:27.632748 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309b4c3b-b659-4a9e-a56d-7e91513aa519 None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:27.633311 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309b4c3b-b659-4a9e-a56d-7e91513aa519 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:27.633783 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309b4c3b-b659-4a9e-a56d-7e91513aa519 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:27.634680 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309b4c3b-b659-4a9e-a56d-7e91513aa519 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:27.636309 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309b4c3b-b659-4a9e-a56d-7e91513aa519 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:27.644076 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:27.653012 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.sql.core [None req-309b4c3b-b659-4a9e-a56d-7e91513aa519 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 30 00:43:27.653012 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 30 00:43:27.653012 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [parameters: {'id': '86c2b9f0309f412bb037b4dfd4d9dd1f', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 30 00:43:27.653012 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81495) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 30 00:43:27.664539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-309b4c3b-b659-4a9e-a56d-7e91513aa519 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 30 00:43:27.664539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:27.664539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 30 00:43:27.664539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 30 00:43:27.664539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 30 00:43:27.664539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 30 00:43:27.664539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 30 00:43:27.664539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application result = self._query(query) Oct 30 00:43:27.664539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 30 00:43:27.664539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application conn.query(q) Oct 30 00:43:27.664539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 30 00:43:27.664539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 30 00:43:27.664539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 30 00:43:27.664539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application result.read() Oct 30 00:43:27.664539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 30 00:43:27.665418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 30 00:43:27.665418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 30 00:43:27.665418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 30 00:43:27.665418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 30 00:43:27.665418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 30 00:43:27.665418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 30 00:43:27.665418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 30 00:43:27.665418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 30 00:43:27.665418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:27.665418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 30 00:43:27.665418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:27.665418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:27.665418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 30 00:43:27.665418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 30 00:43:27.665418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Oct 30 00:43:27.665418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return ref.to_dict() Oct 30 00:43:27.665418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 30 00:43:27.665418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application next(self.gen) Oct 30 00:43:27.666356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 30 00:43:27.666356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application yield resource Oct 30 00:43:27.666356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 30 00:43:27.666356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application next(self.gen) Oct 30 00:43:27.666356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 30 00:43:27.666356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self.session.rollback() Oct 30 00:43:27.666356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 30 00:43:27.666356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self.force_reraise() Oct 30 00:43:27.666356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 30 00:43:27.666356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise self.value Oct 30 00:43:27.666356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 30 00:43:27.666356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 30 00:43:27.666356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 30 00:43:27.666356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application session.commit() Oct 30 00:43:27.666356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 30 00:43:27.666356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 30 00:43:27.666356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 30 00:43:27.667202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._prepare_impl() Oct 30 00:43:27.667202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 30 00:43:27.667202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self.session.flush() Oct 30 00:43:27.667202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 30 00:43:27.667202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._flush(objects) Oct 30 00:43:27.667202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 30 00:43:27.667202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 30 00:43:27.667202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 30 00:43:27.667202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application compat.raise_( Oct 30 00:43:27.667202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 30 00:43:27.667202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception Oct 30 00:43:27.667202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 30 00:43:27.667202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application flush_context.execute() Oct 30 00:43:27.667202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 30 00:43:27.667202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rec.execute(self) Oct 30 00:43:27.667202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 30 00:43:27.667202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 30 00:43:27.668123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 30 00:43:27.668123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 30 00:43:27.668123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 30 00:43:27.668123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 30 00:43:27.668123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 30 00:43:27.668123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 30 00:43:27.668123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 30 00:43:27.668123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 30 00:43:27.668123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 30 00:43:27.668123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 30 00:43:27.668123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 30 00:43:27.668123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 30 00:43:27.668123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 30 00:43:27.668123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 30 00:43:27.668123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 30 00:43:27.668123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception Oct 30 00:43:27.668909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 30 00:43:27.668909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 30 00:43:27.668909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 30 00:43:27.668909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 30 00:43:27.668909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 30 00:43:27.668909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application result = self._query(query) Oct 30 00:43:27.668909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 30 00:43:27.668909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application conn.query(q) Oct 30 00:43:27.668909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 30 00:43:27.668909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 30 00:43:27.668909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 30 00:43:27.668909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application result.read() Oct 30 00:43:27.668909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 30 00:43:27.668909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 30 00:43:27.668909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 30 00:43:27.668909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 30 00:43:27.669703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 30 00:43:27.669703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 30 00:43:27.669703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 30 00:43:27.669703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 30 00:43:27.669703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 30 00:43:27.669703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 30 00:43:27.669703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [parameters: {'id': '86c2b9f0309f412bb037b4dfd4d9dd1f', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 30 00:43:27.669703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 30 00:43:27.669703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:27.669703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:43:27.669703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:27.669703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:27.669703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:27.669703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:27.669703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:27.669703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:27.670487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:27.670487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:27.670487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:27.670487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:27.670487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:27.670487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:27.670487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Oct 30 00:43:27.670487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Oct 30 00:43:27.670487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:27.670487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:27.670487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Oct 30 00:43:27.670487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Oct 30 00:43:27.670487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 30 00:43:27.670487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 30 00:43:27.670487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 30 00:43:27.670487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:27.674559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 179/358] 10.222.0.16 () {60 vars in 1131 bytes} [Sun Oct 30 00:43:27 2022] POST /identity/v3/roles => generated 144 bytes in 205 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:43:27.680095 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309b4c3b-b659-4a9e-a56d-7e91513aa519 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:27.718924 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c738a6-645a-46e3-b4aa-9de66b69068a None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:27.793467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095b880d-00b8-498b-8209-44b3db0ac8c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:27.794084 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095b880d-00b8-498b-8209-44b3db0ac8c6 None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:27.794357 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095b880d-00b8-498b-8209-44b3db0ac8c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:27.794623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095b880d-00b8-498b-8209-44b3db0ac8c6 None admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:27.797433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095b880d-00b8-498b-8209-44b3db0ac8c6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:27.800157 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095b880d-00b8-498b-8209-44b3db0ac8c6 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:27.837754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 180/359] 10.222.0.16 () {60 vars in 1132 bytes} [Sun Oct 30 00:43:27 2022] POST /identity/v3/users => generated 249 bytes in 333 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:43:27.854781 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309b4c3b-b659-4a9e-a56d-7e91513aa519 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:27.898851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc36f21d-b82c-4bf8-ac38-873943e3cb92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:27.899518 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc36f21d-b82c-4bf8-ac38-873943e3cb92 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:27.899859 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc36f21d-b82c-4bf8-ac38-873943e3cb92 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:27.900134 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc36f21d-b82c-4bf8-ac38-873943e3cb92 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:27.918474 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc36f21d-b82c-4bf8-ac38-873943e3cb92 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:27.923582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-dc36f21d-b82c-4bf8-ac38-873943e3cb92 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 30 00:43:27.923582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:27.923582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:27.923582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:27.923582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:27.923582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:27.923582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:27.923582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:27.923582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:27.923582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:27.923582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:27.923582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:27.923582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:27.923582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:27.923582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:27.923582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:27.924702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:27.924702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:27.924702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:27.924702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:27.924702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:27.924702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:27.924702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:27.924702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:27.924702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:27.924702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:27.924702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:27.924702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:27.924702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:27.924702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:27.924702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:27.924702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:27.925626 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:27.925626 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:27.925626 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:27.925626 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:27.925626 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:27.925626 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:27.925626 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:27.925626 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:27.925626 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:27.925626 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:27.925626 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 30 00:43:27.925626 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:27.925626 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 180/360] 10.222.0.16 () {56 vars in 1114 bytes} [Sun Oct 30 00:43:27 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:27.931344 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095b880d-00b8-498b-8209-44b3db0ac8c6 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:27.964797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:28.092968 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095b880d-00b8-498b-8209-44b3db0ac8c6 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:28.138078 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81e5d64-23cc-41f2-83e8-92e81c7e2c18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:28.139116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81e5d64-23cc-41f2-83e8-92e81c7e2c18 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:28.139577 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81e5d64-23cc-41f2-83e8-92e81c7e2c18 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:28.140238 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81e5d64-23cc-41f2-83e8-92e81c7e2c18 None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:28.141092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81e5d64-23cc-41f2-83e8-92e81c7e2c18 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:28.141577 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81e5d64-23cc-41f2-83e8-92e81c7e2c18 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:28.143171 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81e5d64-23cc-41f2-83e8-92e81c7e2c18 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:28.153616 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 181/361] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:43:27 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:29.236379 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aed8e51-f814-4f37-861e-5aefaf9d90ca None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:29.236919 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aed8e51-f814-4f37-861e-5aefaf9d90ca None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:29.236919 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aed8e51-f814-4f37-861e-5aefaf9d90ca None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:29.238584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 181/362] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:29.243489 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c162f1-338e-4ce1-8362-511a5b1ed043 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:29.243913 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c162f1-338e-4ce1-8362-511a5b1ed043 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:29.244077 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c162f1-338e-4ce1-8362-511a5b1ed043 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:29.272747 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-e5c162f1-338e-4ce1-8362-511a5b1ed043 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:29.273959 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-e5c162f1-338e-4ce1-8362-511a5b1ed043 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:29.364831 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 182/363] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:43:29.704404 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4278c1d-439c-43e4-81ee-0689737c0546 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:29.704788 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4278c1d-439c-43e4-81ee-0689737c0546 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:29.704962 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4278c1d-439c-43e4-81ee-0689737c0546 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:29.706560 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 182/364] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:29.712103 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ac0fe9-216f-431d-84f9-e7da19d4deae None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:29.712385 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ac0fe9-216f-431d-84f9-e7da19d4deae None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:29.712692 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ac0fe9-216f-431d-84f9-e7da19d4deae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:29.727902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:29.736921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-a5ac0fe9-216f-431d-84f9-e7da19d4deae None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:29.738658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a5ac0fe9-216f-431d-84f9-e7da19d4deae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:29.749134 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4278c1d-439c-43e4-81ee-0689737c0546 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:29.851628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:29.862541 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 183/365] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:43:29.871242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b63773-4a11-455c-95f2-fbf01276920d None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:29.871514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b63773-4a11-455c-95f2-fbf01276920d None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:29.871811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b63773-4a11-455c-95f2-fbf01276920d None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:29.873357 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 184/366] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:29.878057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf68e16e-af8a-4126-8eb8-b0d7e8d5de71 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:29.878327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf68e16e-af8a-4126-8eb8-b0d7e8d5de71 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:29.878590 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf68e16e-af8a-4126-8eb8-b0d7e8d5de71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:29.905240 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-bf68e16e-af8a-4126-8eb8-b0d7e8d5de71 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:29.906411 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-bf68e16e-af8a-4126-8eb8-b0d7e8d5de71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:29.937175 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4278c1d-439c-43e4-81ee-0689737c0546 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:29.989167 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 185/367] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:43:30.002416 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ba5aea-b54c-466d-8a43-44364c57f403 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:30.003372 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ba5aea-b54c-466d-8a43-44364c57f403 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:30.003880 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ba5aea-b54c-466d-8a43-44364c57f403 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:30.004346 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ba5aea-b54c-466d-8a43-44364c57f403 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:30.015368 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ba5aea-b54c-466d-8a43-44364c57f403 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:30.020985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-26ba5aea-b54c-466d-8a43-44364c57f403 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:30.020985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:30.020985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:30.020985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:30.020985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:30.020985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:30.020985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:30.020985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:30.020985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:30.020985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:30.020985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:30.020985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:30.020985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:30.020985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:30.020985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:30.020985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:30.021862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:30.021862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:30.021862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:30.021862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:30.021862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:30.021862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:30.021862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:30.021862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:30.021862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:30.021862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:30.021862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:30.021862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:30.021862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:30.021862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:30.021862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:30.021862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:30.022684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:30.022684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:30.022684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:30.022684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:30.022684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:30.022684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:30.022684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:30.022684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:30.022684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:30.022684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:30.022684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:30.022684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:30.024962 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 183/368] 10.222.0.16 () {56 vars in 1102 bytes} [Sun Oct 30 00:43:29 2022] GET /identity/v3/roles/service => generated 85 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:43:30.032910 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf68e16e-af8a-4126-8eb8-b0d7e8d5de71 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:30.108312 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:30.153575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf68e16e-af8a-4126-8eb8-b0d7e8d5de71 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:30.175074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ba5aea-b54c-466d-8a43-44364c57f403 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:30.197296 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9367de55-ca06-4d02-921e-95cf9286ca42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:30.197970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9367de55-ca06-4d02-921e-95cf9286ca42 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:30.198293 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9367de55-ca06-4d02-921e-95cf9286ca42 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:30.198567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9367de55-ca06-4d02-921e-95cf9286ca42 None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:30.199231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9367de55-ca06-4d02-921e-95cf9286ca42 None admin] RBAC: Adding query filter params (name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:30.199531 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9367de55-ca06-4d02-921e-95cf9286ca42 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:30.201411 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9367de55-ca06-4d02-921e-95cf9286ca42 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:30.211186 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 186/369] 10.222.0.16 () {56 vars in 1103 bytes} [Sun Oct 30 00:43:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:30.216155 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9367de55-ca06-4d02-921e-95cf9286ca42 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:30.346833 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9367de55-ca06-4d02-921e-95cf9286ca42 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:30.358135 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ba5aea-b54c-466d-8a43-44364c57f403 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:30.377036 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:30.381971 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:30.393602 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab384e49-2f87-49e0-8d21-4993d379a686 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:30.394384 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab384e49-2f87-49e0-8d21-4993d379a686 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:30.394731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab384e49-2f87-49e0-8d21-4993d379a686 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:30.395069 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab384e49-2f87-49e0-8d21-4993d379a686 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:30.397435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a4a84e-55f1-4a5b-96ad-2e9be7bcb4be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:30.398225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a4a84e-55f1-4a5b-96ad-2e9be7bcb4be None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:30.398644 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a4a84e-55f1-4a5b-96ad-2e9be7bcb4be None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:30.399087 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a4a84e-55f1-4a5b-96ad-2e9be7bcb4be None admin] PATH_INFO: `/v3/roles/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:30.407973 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab384e49-2f87-49e0-8d21-4993d379a686 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:30.410264 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a4a84e-55f1-4a5b-96ad-2e9be7bcb4be None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:30.410622 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab384e49-2f87-49e0-8d21-4993d379a686 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:30.415504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-f0a4a84e-55f1-4a5b-96ad-2e9be7bcb4be None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:30.415504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:30.415504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:30.415504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:30.415504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:30.415504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:30.415504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:30.415504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:30.415504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:30.415504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:30.415504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:30.415504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:30.415504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:30.415504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:30.415504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:30.415504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:30.416609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:30.416609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:30.416609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:30.416609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:30.416609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:30.416609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:30.416609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:30.416609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:30.416609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:30.416609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:30.416609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:30.416609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:30.416609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:30.416609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:30.416609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:30.416609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:30.417624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:30.417624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:30.417624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:30.417624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:30.417624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:30.417624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:30.417624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:30.417624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:30.417624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:30.417624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:30.417624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:30.417624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:30.419992 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 184/370] 10.222.0.16 () {56 vars in 1102 bytes} [Sun Oct 30 00:43:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:43:30.420941 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a4a84e-55f1-4a5b-96ad-2e9be7bcb4be None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:30.429601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-ab384e49-2f87-49e0-8d21-4993d379a686 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:30.429601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:30.429601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:30.429601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:30.429601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:30.429601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:30.429601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:30.429601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:30.429601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:30.429601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:30.429601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:30.429601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:30.429601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:30.429601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:30.429601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:30.429601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:30.430659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:30.430659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:30.430659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:30.430659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:30.430659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:30.430659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:30.430659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:30.430659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:30.430659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:30.430659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:30.430659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:30.430659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:30.430659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:30.430659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:30.430659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:30.430659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:30.430659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:30.431492 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:30.431492 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:30.431492 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:30.431928 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 187/371] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:43:30.437169 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab384e49-2f87-49e0-8d21-4993d379a686 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:30.544927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a4a84e-55f1-4a5b-96ad-2e9be7bcb4be None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:30.577952 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab384e49-2f87-49e0-8d21-4993d379a686 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:30.583246 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16614122-2470-4b6e-b30b-0775f1c29644 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:30.584111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16614122-2470-4b6e-b30b-0775f1c29644 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:30.584571 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16614122-2470-4b6e-b30b-0775f1c29644 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:30.585003 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16614122-2470-4b6e-b30b-0775f1c29644 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:30.596496 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16614122-2470-4b6e-b30b-0775f1c29644 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:30.601862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-16614122-2470-4b6e-b30b-0775f1c29644 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:30.601862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:30.601862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:30.601862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:30.601862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:30.601862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:30.601862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:30.601862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:30.601862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:30.601862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:30.601862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:30.601862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:30.601862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:30.601862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:30.601862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:30.601862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:30.602712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:30.602712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:30.602712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:30.602712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:30.602712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:30.602712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:30.602712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:30.602712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:30.602712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:30.602712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:30.602712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:30.602712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:30.602712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:30.602712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:30.602712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:30.602712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:30.603491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:30.603491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:30.603491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:30.603491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:30.603491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:30.603491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:30.603491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:30.603491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:30.603491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:30.603491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:30.603491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:30.603491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:30.605302 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 185/372] 10.222.0.16 () {56 vars in 1102 bytes} [Sun Oct 30 00:43:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:30.611054 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16614122-2470-4b6e-b30b-0775f1c29644 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:30.613500 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530dc197-b5ff-4b2f-a1a1-bb59b56f89b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:30.614116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530dc197-b5ff-4b2f-a1a1-bb59b56f89b7 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:30.614604 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530dc197-b5ff-4b2f-a1a1-bb59b56f89b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:30.614878 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530dc197-b5ff-4b2f-a1a1-bb59b56f89b7 None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:30.616171 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530dc197-b5ff-4b2f-a1a1-bb59b56f89b7 None admin] RBAC: Adding query filter params (name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:30.616478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530dc197-b5ff-4b2f-a1a1-bb59b56f89b7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:30.617863 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530dc197-b5ff-4b2f-a1a1-bb59b56f89b7 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:30.627685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 188/373] 10.222.0.16 () {56 vars in 1103 bytes} [Sun Oct 30 00:43:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:30.629037 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44be5529-776a-4e01-ba24-c30bff99476e None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:30.629316 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44be5529-776a-4e01-ba24-c30bff99476e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:30.629581 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44be5529-776a-4e01-ba24-c30bff99476e None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:30.630936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 189/374] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:30.631433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44be5529-776a-4e01-ba24-c30bff99476e None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:30.660916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:30.681043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:30.729597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16614122-2470-4b6e-b30b-0775f1c29644 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:30.745598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44be5529-776a-4e01-ba24-c30bff99476e None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:30.766924 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32513d9-a408-467a-8db8-fce3005945c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:30.767538 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32513d9-a408-467a-8db8-fce3005945c1 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:30.767846 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32513d9-a408-467a-8db8-fce3005945c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:30.768203 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32513d9-a408-467a-8db8-fce3005945c1 None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:30.768845 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32513d9-a408-467a-8db8-fce3005945c1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:30.769139 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32513d9-a408-467a-8db8-fce3005945c1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:30.770487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32513d9-a408-467a-8db8-fce3005945c1 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:30.786417 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 186/375] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:30.786628 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2fce03-0f22-4809-9389-fc6153b95cf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:30.786628 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2fce03-0f22-4809-9389-fc6153b95cf1 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:30.786800 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32513d9-a408-467a-8db8-fce3005945c1 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:30.786869 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2fce03-0f22-4809-9389-fc6153b95cf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:30.786869 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2fce03-0f22-4809-9389-fc6153b95cf1 None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:30.786869 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc2fce03-0f22-4809-9389-fc6153b95cf1 None admin] RBAC: Adding query filter params (name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:30.786869 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc2fce03-0f22-4809-9389-fc6153b95cf1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:30.786869 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc2fce03-0f22-4809-9389-fc6153b95cf1 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:30.799093 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 190/376] 10.222.0.16 () {56 vars in 1103 bytes} [Sun Oct 30 00:43:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:30.799093 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe9f428-363a-4c35-9dcb-7500845496c4 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:30.799093 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe9f428-363a-4c35-9dcb-7500845496c4 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:30.799093 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe9f428-363a-4c35-9dcb-7500845496c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:30.815983 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-8fe9f428-363a-4c35-9dcb-7500845496c4 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:30.817150 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-8fe9f428-363a-4c35-9dcb-7500845496c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:30.889722 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 191/377] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:30.890742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe9f428-363a-4c35-9dcb-7500845496c4 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:30.892266 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:30.897152 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32513d9-a408-467a-8db8-fce3005945c1 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:30.941497 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:30.947479 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfeda0b-67c8-4815-a139-fb92f3547df2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:30.948157 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfeda0b-67c8-4815-a139-fb92f3547df2 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:30.948427 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfeda0b-67c8-4815-a139-fb92f3547df2 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:30.948690 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfeda0b-67c8-4815-a139-fb92f3547df2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:30.963579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfeda0b-67c8-4815-a139-fb92f3547df2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:30.965722 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfeda0b-67c8-4815-a139-fb92f3547df2 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:30.985105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-7cfeda0b-67c8-4815-a139-fb92f3547df2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:30.985105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:30.985105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:30.985105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:30.985105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:30.985105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:30.985105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:30.985105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:30.985105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:30.985105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:30.985105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:30.985105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:30.985105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:30.985105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:30.985105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:30.985105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:30.986049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:30.986049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:30.986049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:30.986049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:30.986049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:30.986049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:30.986049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:30.986049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:30.986049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:30.986049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:30.986049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:30.986049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:30.986049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:30.986049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:30.986049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:30.986049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:30.986049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:30.987074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:30.987074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:30.987074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:30.987074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 187/378] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:30.987799 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfeda0b-67c8-4815-a139-fb92f3547df2 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:31.023637 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe9f428-363a-4c35-9dcb-7500845496c4 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:31.061066 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56c79e2-0200-4259-b311-ab597c59e005 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:31.061696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56c79e2-0200-4259-b311-ab597c59e005 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:31.061972 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56c79e2-0200-4259-b311-ab597c59e005 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:31.062237 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56c79e2-0200-4259-b311-ab597c59e005 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:31.069441 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56c79e2-0200-4259-b311-ab597c59e005 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:31.070885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56c79e2-0200-4259-b311-ab597c59e005 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:31.084654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-b56c79e2-0200-4259-b311-ab597c59e005 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 30 00:43:31.084654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:31.084654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:31.084654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:31.084654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:31.084654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:31.084654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:31.084654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:31.084654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:31.084654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:31.084654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:31.084654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:31.084654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:43:31.084654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:43:31.084654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:43:31.084654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:43:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:43:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:43:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:31.085584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:31.086471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:31.086471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:31.086471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:43:31.086471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:43:31.086471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:43:31.086471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:43:31.086471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:43:31.086471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:43:31.086471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 30 00:43:31.086471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:31.086471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 192/379] 10.222.0.16 () {56 vars in 1093 bytes} [Sun Oct 30 00:43:30 2022] GET /identity/v3/users/nova => generated 82 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:31.087010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56c79e2-0200-4259-b311-ab597c59e005 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:31.106990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfeda0b-67c8-4815-a139-fb92f3547df2 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:31.142207 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c39d26-5a2e-4429-8527-63a93050bc22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:31.142746 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c39d26-5a2e-4429-8527-63a93050bc22 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:31.143072 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c39d26-5a2e-4429-8527-63a93050bc22 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:31.143327 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c39d26-5a2e-4429-8527-63a93050bc22 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:31.153683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c39d26-5a2e-4429-8527-63a93050bc22 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:31.155123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c39d26-5a2e-4429-8527-63a93050bc22 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:31.170323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-a6c39d26-5a2e-4429-8527-63a93050bc22 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:31.170323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:31.170323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:31.170323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:31.170323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:31.170323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:31.170323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:31.170323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:31.170323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:31.170323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:31.170323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:31.170323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:31.170323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:31.170323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:31.170323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:31.170323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:31.171248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:31.171248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:31.171248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:31.171248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:31.171248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:31.171248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:31.171248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:31.171248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:31.171248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:31.171248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:31.171248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:31.171248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:31.171248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:31.171248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:31.171248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:31.171248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:31.171248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:31.172806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:31.172806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:31.172806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:31.172806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 188/380] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:31.172806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98cbd2e9-93e1-4c8b-ad04-4696d66af0eb None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:31.173075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98cbd2e9-93e1-4c8b-ad04-4696d66af0eb None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:31.173383 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98cbd2e9-93e1-4c8b-ad04-4696d66af0eb None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:31.176608 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 189/381] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:31.176608 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98cbd2e9-93e1-4c8b-ad04-4696d66af0eb None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:31.178488 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:31.188690 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:31.196448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56c79e2-0200-4259-b311-ab597c59e005 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:31.231004 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25e84b4-e8d1-4143-8a2d-76f4a4838347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:31.231608 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25e84b4-e8d1-4143-8a2d-76f4a4838347 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:31.231910 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25e84b4-e8d1-4143-8a2d-76f4a4838347 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:31.232177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25e84b4-e8d1-4143-8a2d-76f4a4838347 None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:31.232820 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25e84b4-e8d1-4143-8a2d-76f4a4838347 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:31.233118 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25e84b4-e8d1-4143-8a2d-76f4a4838347 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:31.234494 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25e84b4-e8d1-4143-8a2d-76f4a4838347 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:31.246101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 193/382] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:31.248531 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25e84b4-e8d1-4143-8a2d-76f4a4838347 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:31.278806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98cbd2e9-93e1-4c8b-ad04-4696d66af0eb None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:31.315492 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c510fe9-b3f4-4376-b9eb-f32a1627b610 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:31.316126 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c510fe9-b3f4-4376-b9eb-f32a1627b610 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:31.316398 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c510fe9-b3f4-4376-b9eb-f32a1627b610 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:31.316662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c510fe9-b3f4-4376-b9eb-f32a1627b610 None admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:31.317380 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c510fe9-b3f4-4376-b9eb-f32a1627b610 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:31.317675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c510fe9-b3f4-4376-b9eb-f32a1627b610 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:31.319095 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c510fe9-b3f4-4376-b9eb-f32a1627b610 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:31.348524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25e84b4-e8d1-4143-8a2d-76f4a4838347 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:31.357129 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 190/383] 10.222.0.16 () {56 vars in 1133 bytes} [Sun Oct 30 00:43:31 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:31.357966 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c510fe9-b3f4-4376-b9eb-f32a1627b610 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:31.387602 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d15f63-1c59-47e3-a1d1-3c0d99115b30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:31.388261 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d15f63-1c59-47e3-a1d1-3c0d99115b30 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:31.388538 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d15f63-1c59-47e3-a1d1-3c0d99115b30 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:31.388809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d15f63-1c59-47e3-a1d1-3c0d99115b30 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:31.399947 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d15f63-1c59-47e3-a1d1-3c0d99115b30 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:31.404702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-51d15f63-1c59-47e3-a1d1-3c0d99115b30 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:31.404702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:31.404702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:31.404702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:31.404702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:31.404702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:31.404702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:31.404702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:31.404702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:31.404702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:31.404702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:31.404702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:31.404702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:31.404702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:31.404702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:31.404702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:31.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:31.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:31.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:31.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:31.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:31.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:31.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:31.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:31.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:31.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:31.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:31.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:31.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:31.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:31.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:31.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:31.408829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:31.408829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:31.408829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:31.408829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:31.408829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:31.408829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:31.408829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:31.408829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:31.408829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:31.408829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:31.408829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:31.408829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:31.408829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 194/384] 10.222.0.16 () {56 vars in 1102 bytes} [Sun Oct 30 00:43:31 2022] GET /identity/v3/roles/service => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:31.408829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bee4d3a-c99b-4d83-bfd3-d1c6f6434ffd None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:31.409656 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bee4d3a-c99b-4d83-bfd3-d1c6f6434ffd None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:31.409656 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bee4d3a-c99b-4d83-bfd3-d1c6f6434ffd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:31.416504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:31.426245 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-4bee4d3a-c99b-4d83-bfd3-d1c6f6434ffd None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:31.427439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-4bee4d3a-c99b-4d83-bfd3-d1c6f6434ffd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:31.437322 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:31.456200 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c510fe9-b3f4-4376-b9eb-f32a1627b610 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:31.489153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6a9acb-fed0-44e9-8f31-f6ae4d394c73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:31.489750 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6a9acb-fed0-44e9-8f31-f6ae4d394c73 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:31.490017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6a9acb-fed0-44e9-8f31-f6ae4d394c73 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:31.490279 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6a9acb-fed0-44e9-8f31-f6ae4d394c73 None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:31.490965 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6a9acb-fed0-44e9-8f31-f6ae4d394c73 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:31.491258 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6a9acb-fed0-44e9-8f31-f6ae4d394c73 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:31.492648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6a9acb-fed0-44e9-8f31-f6ae4d394c73 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:31.494553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 195/385] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:31.495380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bee4d3a-c99b-4d83-bfd3-d1c6f6434ffd None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:31.508579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 191/386] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:31.508579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6a9acb-fed0-44e9-8f31-f6ae4d394c73 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:31.599095 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bee4d3a-c99b-4d83-bfd3-d1c6f6434ffd None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:31.610240 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6a9acb-fed0-44e9-8f31-f6ae4d394c73 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:31.630147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:31.633057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e37493-660d-4a0a-a000-3529c8f198e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:31.634241 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e37493-660d-4a0a-a000-3529c8f198e6 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:31.634842 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e37493-660d-4a0a-a000-3529c8f198e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:31.635346 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e37493-660d-4a0a-a000-3529c8f198e6 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:31.644082 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e2743a-aebb-4c9f-9d5a-f10675e40c18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:31.644697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e2743a-aebb-4c9f-9d5a-f10675e40c18 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:31.644986 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e2743a-aebb-4c9f-9d5a-f10675e40c18 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:31.645238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e2743a-aebb-4c9f-9d5a-f10675e40c18 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:31.645692 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e37493-660d-4a0a-a000-3529c8f198e6 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:31.647468 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e37493-660d-4a0a-a000-3529c8f198e6 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:31.655598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e2743a-aebb-4c9f-9d5a-f10675e40c18 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:31.657013 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e2743a-aebb-4c9f-9d5a-f10675e40c18 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:31.661254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-e1e37493-660d-4a0a-a000-3529c8f198e6 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 30 00:43:31.661254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:31.661254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:31.661254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:31.661254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:31.661254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:31.661254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:31.661254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:31.661254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:31.661254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:31.661254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:31.661254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:31.661254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:43:31.661254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:43:31.661254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:43:31.661254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:43:31.662306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:31.662306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:31.662306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:43:31.662306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:31.662306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:43:31.662306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:31.662306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:31.662306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:31.662306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:31.662306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:31.662306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:31.662306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:31.662306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:31.662306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:31.662306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:31.662306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:31.662306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:31.662306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:31.663392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:31.663392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:31.663392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:43:31.663392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:43:31.663392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:43:31.663392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:43:31.663392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:43:31.663392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:43:31.663392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 30 00:43:31.663392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:31.665095 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 196/387] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:43:31 2022] GET /identity/v3/users/cinder => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:31.665694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e37493-660d-4a0a-a000-3529c8f198e6 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:31.672599 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:31.680884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-b3e2743a-aebb-4c9f-9d5a-f10675e40c18 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:31.680884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:31.680884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:31.680884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:31.680884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:31.680884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:31.680884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:31.680884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:31.680884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:31.680884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:31.680884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:31.680884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:31.680884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:31.680884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:31.680884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:31.680884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:31.681955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:31.681955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:31.681955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:31.681955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:31.681955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:31.681955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:31.681955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:31.681955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:31.681955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:31.681955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:31.681955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:31.681955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:31.681955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:31.681955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:31.681955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:31.681955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:31.681955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:31.682978 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:31.682978 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:31.682978 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:31.686820 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 192/388] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:31.687601 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e2743a-aebb-4c9f-9d5a-f10675e40c18 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:31.775106 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e37493-660d-4a0a-a000-3529c8f198e6 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:31.788849 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e2743a-aebb-4c9f-9d5a-f10675e40c18 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:31.807692 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db3d8a1-7ad8-4b46-b211-5b0c85895ffc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:31.808587 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db3d8a1-7ad8-4b46-b211-5b0c85895ffc None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:31.808870 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db3d8a1-7ad8-4b46-b211-5b0c85895ffc None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:31.809158 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db3d8a1-7ad8-4b46-b211-5b0c85895ffc None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:31.809815 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db3d8a1-7ad8-4b46-b211-5b0c85895ffc None admin] RBAC: Adding query filter params (name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:31.810097 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db3d8a1-7ad8-4b46-b211-5b0c85895ffc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:31.811807 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db3d8a1-7ad8-4b46-b211-5b0c85895ffc None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:31.819581 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 197/389] 10.222.0.16 () {56 vars in 1103 bytes} [Sun Oct 30 00:43:31 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:31.820463 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db3d8a1-7ad8-4b46-b211-5b0c85895ffc None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:31.821862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77e08be-db21-47ec-82be-7d095ca24984 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:31.822478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77e08be-db21-47ec-82be-7d095ca24984 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:31.822737 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77e08be-db21-47ec-82be-7d095ca24984 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:31.822996 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77e08be-db21-47ec-82be-7d095ca24984 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:31.830814 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77e08be-db21-47ec-82be-7d095ca24984 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:31.832497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77e08be-db21-47ec-82be-7d095ca24984 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:31.844514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-c77e08be-db21-47ec-82be-7d095ca24984 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 30 00:43:31.844514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:31.844514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:31.844514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:31.844514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:31.844514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:31.844514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:31.844514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:31.844514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:31.844514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:31.844514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:31.844514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:31.844514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:43:31.844514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:43:31.844514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:43:31.844514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:43:31.845584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:31.845584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:31.845584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:43:31.845584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:31.845584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:43:31.845584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:31.845584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:31.845584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:31.845584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:31.845584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:31.845584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:31.845584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:31.845584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:31.845584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:31.845584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:31.845584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:31.845584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:31.845584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:31.847460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:31.847460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:31.847460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:43:31.847460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:43:31.847460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:43:31.847460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:43:31.847460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:43:31.847460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:43:31.847460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 30 00:43:31.847460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:31.847460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 193/390] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:43:31 2022] GET /identity/v3/users/glance => generated 84 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:31.847460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77e08be-db21-47ec-82be-7d095ca24984 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:31.878498 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:31.888942 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:31.923250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db3d8a1-7ad8-4b46-b211-5b0c85895ffc None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:31.945164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77e08be-db21-47ec-82be-7d095ca24984 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:31.955246 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a578059e-2e0c-413b-9a36-3adcc547d594 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:31.956094 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a578059e-2e0c-413b-9a36-3adcc547d594 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:31.956368 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a578059e-2e0c-413b-9a36-3adcc547d594 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:31.956635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a578059e-2e0c-413b-9a36-3adcc547d594 None admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:31.957361 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a578059e-2e0c-413b-9a36-3adcc547d594 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:31.957654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a578059e-2e0c-413b-9a36-3adcc547d594 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:31.959062 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a578059e-2e0c-413b-9a36-3adcc547d594 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:31.977972 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8436732e-15b4-4acd-8831-b3ad2c611f0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:31.978559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8436732e-15b4-4acd-8831-b3ad2c611f0b None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:31.978828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8436732e-15b4-4acd-8831-b3ad2c611f0b None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:31.979091 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8436732e-15b4-4acd-8831-b3ad2c611f0b None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:31.979731 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8436732e-15b4-4acd-8831-b3ad2c611f0b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:31.980030 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8436732e-15b4-4acd-8831-b3ad2c611f0b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:31.981382 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8436732e-15b4-4acd-8831-b3ad2c611f0b None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:31.993597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 194/391] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:31.994382 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8436732e-15b4-4acd-8831-b3ad2c611f0b None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:31.995393 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 198/392] 10.222.0.16 () {56 vars in 1137 bytes} [Sun Oct 30 00:43:31 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:31.996258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a578059e-2e0c-413b-9a36-3adcc547d594 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:32.089165 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:32.098951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8436732e-15b4-4acd-8831-b3ad2c611f0b None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:32.099196 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a578059e-2e0c-413b-9a36-3adcc547d594 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:32.123307 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:32.134178 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2ce1ea-179f-4344-a122-a6dada016a8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:32.134688 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e428317f-7ba0-472a-9820-a93f8c00a1d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:32.134822 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2ce1ea-179f-4344-a122-a6dada016a8c None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:32.135066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2ce1ea-179f-4344-a122-a6dada016a8c None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:32.135311 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e428317f-7ba0-472a-9820-a93f8c00a1d1 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:32.135399 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2ce1ea-179f-4344-a122-a6dada016a8c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:32.135592 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e428317f-7ba0-472a-9820-a93f8c00a1d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:32.135896 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e428317f-7ba0-472a-9820-a93f8c00a1d1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:32.145073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2ce1ea-179f-4344-a122-a6dada016a8c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:32.145702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e428317f-7ba0-472a-9820-a93f8c00a1d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:32.146514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2ce1ea-179f-4344-a122-a6dada016a8c None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:32.147142 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e428317f-7ba0-472a-9820-a93f8c00a1d1 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:32.160905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-fe2ce1ea-179f-4344-a122-a6dada016a8c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:32.160905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:32.160905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:32.160905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:32.160905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:32.160905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:32.160905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:32.160905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:32.160905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:32.160905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:32.160905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:32.160905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:32.160905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:32.160905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:32.160905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:32.160905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:32.161927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:32.161927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:32.161927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:32.161927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:32.161927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:32.161927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:32.161927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:32.161927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:32.161927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:32.161927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:32.161927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:32.161927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:32.161927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:32.161927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:32.161927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:32.161927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:32.161927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:32.164688 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:32.164688 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:32.164688 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:32.164850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-e428317f-7ba0-472a-9820-a93f8c00a1d1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:32.164850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:32.164850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:32.164850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:32.164850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:32.164850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:32.164850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:32.164850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:32.164850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:32.164850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:32.164850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:32.164850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:32.164850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:32.164850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:32.164850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:32.164850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:32.165627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:32.165627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:32.165627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:32.165627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:32.165627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:32.165627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:32.165627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:32.165627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:32.165627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:32.165627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:32.165627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:32.165627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:32.165627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:32.165627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:32.165627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:32.165627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:32.165627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:32.166485 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:32.166485 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:32.166485 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:32.166640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 195/393] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:32.166640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2ce1ea-179f-4344-a122-a6dada016a8c None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:32.166773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 199/394] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:32.166773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e428317f-7ba0-472a-9820-a93f8c00a1d1 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:32.262649 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2ce1ea-179f-4344-a122-a6dada016a8c None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:32.264459 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e428317f-7ba0-472a-9820-a93f8c00a1d1 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:32.295688 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e7b844-b04d-4288-8dc4-888c289c3259 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:32.296306 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e7b844-b04d-4288-8dc4-888c289c3259 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:32.296579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e7b844-b04d-4288-8dc4-888c289c3259 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:32.296846 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e7b844-b04d-4288-8dc4-888c289c3259 None admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:32.297307 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7d8880-3006-4462-8dda-a1c355e72d55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:32.297556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e7b844-b04d-4288-8dc4-888c289c3259 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:32.297854 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e7b844-b04d-4288-8dc4-888c289c3259 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:32.297949 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7d8880-3006-4462-8dda-a1c355e72d55 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:32.298180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7d8880-3006-4462-8dda-a1c355e72d55 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:32.298446 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7d8880-3006-4462-8dda-a1c355e72d55 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:32.299356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e7b844-b04d-4288-8dc4-888c289c3259 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:32.304941 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7d8880-3006-4462-8dda-a1c355e72d55 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:32.306364 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7d8880-3006-4462-8dda-a1c355e72d55 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:32.316562 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 196/395] 10.222.0.16 () {56 vars in 1137 bytes} [Sun Oct 30 00:43:32 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:32.317352 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e7b844-b04d-4288-8dc4-888c289c3259 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:32.318702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-4f7d8880-3006-4462-8dda-a1c355e72d55 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:32.318702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:32.318702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:32.318702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:32.318702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:32.318702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:32.318702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:32.318702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:32.318702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:32.318702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:32.318702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:32.318702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:32.318702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:43:32.318702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:43:32.318702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:43:32.319707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:43:32.319707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:32.319707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:32.319707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:32.319707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:32.319707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:32.319707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:32.319707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:32.319707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:32.319707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:32.319707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:32.319707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:32.319707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:32.319707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:32.319707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:32.319707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:32.319707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:32.325520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:43:32.325520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:43:32.325520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:43:32.325520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:43:32.325520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:43:32.325520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:43:32.325520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:32.325520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:32.325520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 200/396] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:43:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:32.325520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7d8880-3006-4462-8dda-a1c355e72d55 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:32.327168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:32.362324 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:32.421747 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e7b844-b04d-4288-8dc4-888c289c3259 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:32.425285 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7d8880-3006-4462-8dda-a1c355e72d55 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:32.455618 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282f721c-6641-4b33-b557-94a2a94d8319 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:32.456289 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282f721c-6641-4b33-b557-94a2a94d8319 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:32.456562 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282f721c-6641-4b33-b557-94a2a94d8319 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:32.456829 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282f721c-6641-4b33-b557-94a2a94d8319 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:32.457770 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4675ab25-ba33-4c1b-9d90-3ac59dbecab8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:32.458431 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4675ab25-ba33-4c1b-9d90-3ac59dbecab8 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:32.458771 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4675ab25-ba33-4c1b-9d90-3ac59dbecab8 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:32.459100 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4675ab25-ba33-4c1b-9d90-3ac59dbecab8 None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:32.459827 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4675ab25-ba33-4c1b-9d90-3ac59dbecab8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:32.460184 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4675ab25-ba33-4c1b-9d90-3ac59dbecab8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:32.461614 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4675ab25-ba33-4c1b-9d90-3ac59dbecab8 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:32.466728 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282f721c-6641-4b33-b557-94a2a94d8319 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:32.468323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282f721c-6641-4b33-b557-94a2a94d8319 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:32.472962 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 201/397] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:32.473809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4675ab25-ba33-4c1b-9d90-3ac59dbecab8 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:32.484024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-282f721c-6641-4b33-b557-94a2a94d8319 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:32.484024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:32.484024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:32.484024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:32.484024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:32.484024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:32.484024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:32.484024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:32.484024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:32.484024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:32.484024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:32.484024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:32.484024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:32.484024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:32.484024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:32.484024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:32.485074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:32.485074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:32.485074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:32.485074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:32.485074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:32.485074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:32.485074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:32.485074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:32.485074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:32.485074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:32.485074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:32.485074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:32.485074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:32.485074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:32.485074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:32.485074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:32.485074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:32.486135 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:32.486135 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:32.486135 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:32.489685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 197/398] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:32.490475 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282f721c-6641-4b33-b557-94a2a94d8319 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:32.541840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:32.564645 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:32.570412 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4675ab25-ba33-4c1b-9d90-3ac59dbecab8 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:32.587894 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282f721c-6641-4b33-b557-94a2a94d8319 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:32.603121 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db482cc9-533c-4b3d-9693-049c60c74411 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:32.603744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db482cc9-533c-4b3d-9693-049c60c74411 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:32.604024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db482cc9-533c-4b3d-9693-049c60c74411 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:32.604290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db482cc9-533c-4b3d-9693-049c60c74411 None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:32.604920 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db482cc9-533c-4b3d-9693-049c60c74411 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:32.605219 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db482cc9-533c-4b3d-9693-049c60c74411 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:32.606576 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db482cc9-533c-4b3d-9693-049c60c74411 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:32.617611 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 202/399] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:32.618388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db482cc9-533c-4b3d-9693-049c60c74411 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:32.619391 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719ba180-6817-4513-818f-5381c60cd843 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:32.620022 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719ba180-6817-4513-818f-5381c60cd843 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:32.620294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719ba180-6817-4513-818f-5381c60cd843 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:32.620555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719ba180-6817-4513-818f-5381c60cd843 None admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:32.621227 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719ba180-6817-4513-818f-5381c60cd843 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:32.621519 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719ba180-6817-4513-818f-5381c60cd843 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:32.622938 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719ba180-6817-4513-818f-5381c60cd843 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:32.641506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 198/400] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:43:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:32.642192 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719ba180-6817-4513-818f-5381c60cd843 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:32.721070 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db482cc9-533c-4b3d-9693-049c60c74411 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:32.740779 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719ba180-6817-4513-818f-5381c60cd843 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:32.754393 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:32.757747 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a32890-330f-4363-8dce-685b63bee6be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:32.758372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a32890-330f-4363-8dce-685b63bee6be None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:32.758645 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a32890-330f-4363-8dce-685b63bee6be None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:32.758905 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a32890-330f-4363-8dce-685b63bee6be None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:32.769564 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a32890-330f-4363-8dce-685b63bee6be None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:32.771008 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a32890-330f-4363-8dce-685b63bee6be None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:32.776739 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f46eb2c-625e-401e-ba8d-18a111438f36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:32.777733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f46eb2c-625e-401e-ba8d-18a111438f36 None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:32.778010 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f46eb2c-625e-401e-ba8d-18a111438f36 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:32.778275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f46eb2c-625e-401e-ba8d-18a111438f36 None admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:32.779291 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f46eb2c-625e-401e-ba8d-18a111438f36 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:32.780737 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f46eb2c-625e-401e-ba8d-18a111438f36 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:32.785071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-d1a32890-330f-4363-8dce-685b63bee6be None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:32.785071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:32.785071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:32.785071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:32.785071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:32.785071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:32.785071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:32.785071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:32.785071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:32.785071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:32.785071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:32.785071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:32.785071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:32.785071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:32.785071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:32.785071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:32.786135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:32.786135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:32.786135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:32.786135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:32.786135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:32.786135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:32.786135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:32.786135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:32.786135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:32.786135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:32.786135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:32.786135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:32.786135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:32.786135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:32.786135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:32.786135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:32.786135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:32.787006 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:32.787006 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:32.787006 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:32.787006 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 203/401] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:32.787006 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a32890-330f-4363-8dce-685b63bee6be None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:32.799708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:32.803593 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 199/402] 10.222.0.16 () {60 vars in 1132 bytes} [Sun Oct 30 00:43:32 2022] POST /identity/v3/users => generated 247 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:43:32.804433 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f46eb2c-625e-401e-ba8d-18a111438f36 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:32.918757 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a32890-330f-4363-8dce-685b63bee6be None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:32.918977 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f46eb2c-625e-401e-ba8d-18a111438f36 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:32.957151 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63478ec3-257e-4095-8c3a-c7ac1473d436 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:32.957796 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63478ec3-257e-4095-8c3a-c7ac1473d436 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:32.958064 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63478ec3-257e-4095-8c3a-c7ac1473d436 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:32.958358 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63478ec3-257e-4095-8c3a-c7ac1473d436 None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:32.958977 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904fa3c7-c7b5-4194-a908-be8af89c4593 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:32.959099 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63478ec3-257e-4095-8c3a-c7ac1473d436 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:32.959291 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63478ec3-257e-4095-8c3a-c7ac1473d436 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:32.960031 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904fa3c7-c7b5-4194-a908-be8af89c4593 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:32.960457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904fa3c7-c7b5-4194-a908-be8af89c4593 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:32.960739 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63478ec3-257e-4095-8c3a-c7ac1473d436 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:32.961004 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904fa3c7-c7b5-4194-a908-be8af89c4593 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:32.970212 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904fa3c7-c7b5-4194-a908-be8af89c4593 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:32.971781 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904fa3c7-c7b5-4194-a908-be8af89c4593 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:32.975538 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 204/403] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:32.976551 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63478ec3-257e-4095-8c3a-c7ac1473d436 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:32.983682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-904fa3c7-c7b5-4194-a908-be8af89c4593 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 30 00:43:32.983682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:32.983682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:32.983682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:32.983682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:32.983682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:32.983682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:32.983682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:32.983682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:32.983682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:32.983682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:32.983682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:32.983682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:43:32.983682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:43:32.983682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:43:32.983682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:43:32.985366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:32.985366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:32.985366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:43:32.985366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:32.985366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:43:32.985366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:32.985366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:32.985366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:32.985366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:32.985366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:32.985366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:32.985366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:32.985366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:32.985366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:32.985366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:32.985366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:32.985366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:32.985366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:32.986246 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:32.986246 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:32.986246 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:43:32.986246 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:43:32.986246 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:43:32.986246 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:43:32.986246 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:43:32.986246 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:43:32.986246 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 30 00:43:32.986246 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:32.986246 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 200/404] 10.222.0.16 () {56 vars in 1102 bytes} [Sun Oct 30 00:43:32 2022] GET /identity/v3/users/neutron => generated 85 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:32.986246 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904fa3c7-c7b5-4194-a908-be8af89c4593 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:33.019174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:33.055509 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:33.090183 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63478ec3-257e-4095-8c3a-c7ac1473d436 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:33.093410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904fa3c7-c7b5-4194-a908-be8af89c4593 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:33.124803 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96c88cd-f9a1-472b-9c23-4e4b208ee503 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:33.125616 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96c88cd-f9a1-472b-9c23-4e4b208ee503 None admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:33.125975 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96c88cd-f9a1-472b-9c23-4e4b208ee503 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:33.126387 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96c88cd-f9a1-472b-9c23-4e4b208ee503 None admin] PATH_INFO: `/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/e7a886715b5b4c1d99f0b37c58cb5417/roles/5a0f6f642a10411da6fc1bfc92a7a3d3` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:33.126992 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc11875f-9c33-45ad-be1a-0d747717c808 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:33.127635 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc11875f-9c33-45ad-be1a-0d747717c808 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:33.129683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc11875f-9c33-45ad-be1a-0d747717c808 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:33.129683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc11875f-9c33-45ad-be1a-0d747717c808 None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:33.129683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc11875f-9c33-45ad-be1a-0d747717c808 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:33.129683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc11875f-9c33-45ad-be1a-0d747717c808 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:33.130410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc11875f-9c33-45ad-be1a-0d747717c808 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:33.146743 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 201/405] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:33.146743 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc11875f-9c33-45ad-be1a-0d747717c808 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:33.147950 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96c88cd-f9a1-472b-9c23-4e4b208ee503 None admin] RBAC: Authorizing `identity:create_grant(project_id=658799ed31d2472fb36af4aff0f03aab, user_id=e7a886715b5b4c1d99f0b37c58cb5417, role_id=5a0f6f642a10411da6fc1bfc92a7a3d3)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:33.149480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96c88cd-f9a1-472b-9c23-4e4b208ee503 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:33.164165 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 205/406] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:43:32 2022] PUT /identity/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/e7a886715b5b4c1d99f0b37c58cb5417/roles/5a0f6f642a10411da6fc1bfc92a7a3d3 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:43:33.172939 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96c88cd-f9a1-472b-9c23-4e4b208ee503 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:33.250018 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:33.260897 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc11875f-9c33-45ad-be1a-0d747717c808 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:33.296585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96c88cd-f9a1-472b-9c23-4e4b208ee503 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:33.297524 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d6c767-4609-4e9f-a7fb-5b39f1c07bd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:33.298170 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d6c767-4609-4e9f-a7fb-5b39f1c07bd7 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:33.298440 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d6c767-4609-4e9f-a7fb-5b39f1c07bd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:33.298759 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d6c767-4609-4e9f-a7fb-5b39f1c07bd7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:33.307122 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d6c767-4609-4e9f-a7fb-5b39f1c07bd7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:33.308483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d6c767-4609-4e9f-a7fb-5b39f1c07bd7 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:33.321437 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:33.321749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-71d6c767-4609-4e9f-a7fb-5b39f1c07bd7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:33.321749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:33.321749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:33.321749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:33.321749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:33.321749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:33.321749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:33.321749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:33.321749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:33.321749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:33.321749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:33.321749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:33.321749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:43:33.321749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:43:33.321749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:43:33.322597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:43:33.322597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:33.322597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:33.322597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:33.322597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:33.322597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:33.322597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:33.322597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:33.322597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:33.322597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:33.322597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:33.322597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:33.322597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:33.322597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:33.322597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:33.322597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:33.322597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:33.323837 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:43:33.323837 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:43:33.323837 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:43:33.323837 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:43:33.323837 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:43:33.323837 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:43:33.323837 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:33.323837 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:33.323837 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 202/407] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:43:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:33.328705 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d6c767-4609-4e9f-a7fb-5b39f1c07bd7 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:33.347825 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19e5d8d-886a-4963-8793-12cf7db447f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:33.348863 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19e5d8d-886a-4963-8793-12cf7db447f0 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:33.349005 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19e5d8d-886a-4963-8793-12cf7db447f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:33.349366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19e5d8d-886a-4963-8793-12cf7db447f0 None admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:33.350261 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19e5d8d-886a-4963-8793-12cf7db447f0 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:33.351766 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19e5d8d-886a-4963-8793-12cf7db447f0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:33.353527 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19e5d8d-886a-4963-8793-12cf7db447f0 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:33.373698 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 206/408] 10.222.0.16 () {56 vars in 1139 bytes} [Sun Oct 30 00:43:33 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:33.374495 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19e5d8d-886a-4963-8793-12cf7db447f0 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:33.462393 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d6c767-4609-4e9f-a7fb-5b39f1c07bd7 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:33.499940 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003d8b13-038c-4dd1-bc1a-abb88a97b65c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:33.501091 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003d8b13-038c-4dd1-bc1a-abb88a97b65c None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:33.501902 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003d8b13-038c-4dd1-bc1a-abb88a97b65c None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:33.502367 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003d8b13-038c-4dd1-bc1a-abb88a97b65c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:33.510805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003d8b13-038c-4dd1-bc1a-abb88a97b65c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:33.511873 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19e5d8d-886a-4963-8793-12cf7db447f0 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:33.512716 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003d8b13-038c-4dd1-bc1a-abb88a97b65c None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:33.525821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-003d8b13-038c-4dd1-bc1a-abb88a97b65c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:33.525821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:33.525821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:33.525821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:33.525821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:33.525821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:33.525821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:33.525821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:33.525821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:33.525821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:33.525821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:33.525821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:33.525821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:43:33.525821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:43:33.525821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:43:33.526826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:43:33.526826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:33.526826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:33.526826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:33.526826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:33.526826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:33.526826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:33.526826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:33.526826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:33.526826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:33.526826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:33.526826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:33.526826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:33.526826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:33.526826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:33.526826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:33.526826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:33.527696 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:43:33.527696 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:43:33.527696 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:43:33.527696 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:43:33.527696 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:43:33.527696 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:43:33.527696 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:33.527696 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:33.534378 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 203/409] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:43:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:33.534378 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003d8b13-038c-4dd1-bc1a-abb88a97b65c None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:33.546217 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:33.559976 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d41360-fb39-42bd-a1cb-671dd680c13a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:33.561155 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d41360-fb39-42bd-a1cb-671dd680c13a None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:33.561438 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d41360-fb39-42bd-a1cb-671dd680c13a None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:33.561756 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d41360-fb39-42bd-a1cb-671dd680c13a None admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:33.562461 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d41360-fb39-42bd-a1cb-671dd680c13a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:33.562761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d41360-fb39-42bd-a1cb-671dd680c13a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:33.564837 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d41360-fb39-42bd-a1cb-671dd680c13a None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:33.588800 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 207/410] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:43:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:33.590065 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d41360-fb39-42bd-a1cb-671dd680c13a None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:33.634476 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:33.652255 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003d8b13-038c-4dd1-bc1a-abb88a97b65c None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:33.696249 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1762af-c4df-4e0d-8a97-a2ac58a8787e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:33.697300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1762af-c4df-4e0d-8a97-a2ac58a8787e None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:33.697572 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1762af-c4df-4e0d-8a97-a2ac58a8787e None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:33.697992 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1762af-c4df-4e0d-8a97-a2ac58a8787e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:33.707403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d41360-fb39-42bd-a1cb-671dd680c13a None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:33.711334 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1762af-c4df-4e0d-8a97-a2ac58a8787e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:33.712942 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1762af-c4df-4e0d-8a97-a2ac58a8787e None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:33.732512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-4d1762af-c4df-4e0d-8a97-a2ac58a8787e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:33.732512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:33.732512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:33.732512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:33.732512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:33.732512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:33.732512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:33.732512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:33.732512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:33.732512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:33.732512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:33.732512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:33.732512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:33.732512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:33.732512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:33.732512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:33.733582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:33.733582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:33.733582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:33.733582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:33.733582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:33.733582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:33.733582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:33.733582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:33.733582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:33.733582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:33.733582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:33.733582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:33.733582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:33.733582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:33.733582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:33.733582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:33.733582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:33.734396 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:33.734396 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:33.734396 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:33.734396 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 204/411] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:33.734625 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1762af-c4df-4e0d-8a97-a2ac58a8787e None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:33.755655 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3981096-2e8d-4133-8f7f-90197050f9ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:33.756710 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3981096-2e8d-4133-8f7f-90197050f9ad None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:33.757172 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3981096-2e8d-4133-8f7f-90197050f9ad None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:33.757675 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3981096-2e8d-4133-8f7f-90197050f9ad None admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:33.758674 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3981096-2e8d-4133-8f7f-90197050f9ad None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:33.759161 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3981096-2e8d-4133-8f7f-90197050f9ad None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:33.760947 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3981096-2e8d-4133-8f7f-90197050f9ad None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:33.778966 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 208/412] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:43:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:33.780252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3981096-2e8d-4133-8f7f-90197050f9ad None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:33.799473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:33.843058 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1762af-c4df-4e0d-8a97-a2ac58a8787e None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:33.883228 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:33.884478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bb0b85-69f0-40dc-8610-bca9f86ec33d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:33.885213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bb0b85-69f0-40dc-8610-bca9f86ec33d None admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:33.885756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bb0b85-69f0-40dc-8610-bca9f86ec33d None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:33.886090 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bb0b85-69f0-40dc-8610-bca9f86ec33d None admin] PATH_INFO: `/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/d57b1c923dff462ab9aaf238f95c6b7f/roles/5a0f6f642a10411da6fc1bfc92a7a3d3` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:33.891153 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3981096-2e8d-4133-8f7f-90197050f9ad None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:33.905744 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0bb0b85-69f0-40dc-8610-bca9f86ec33d None admin] RBAC: Authorizing `identity:create_grant(project_id=658799ed31d2472fb36af4aff0f03aab, user_id=d57b1c923dff462ab9aaf238f95c6b7f, role_id=5a0f6f642a10411da6fc1bfc92a7a3d3)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:33.907374 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0bb0b85-69f0-40dc-8610-bca9f86ec33d None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:33.924518 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 205/413] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:43:33 2022] PUT /identity/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/d57b1c923dff462ab9aaf238f95c6b7f/roles/5a0f6f642a10411da6fc1bfc92a7a3d3 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:43:33.925709 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bb0b85-69f0-40dc-8610-bca9f86ec33d None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:33.928386 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e706fa-e664-4779-ad96-16c45d83ebbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:33.929321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e706fa-e664-4779-ad96-16c45d83ebbe None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:33.929969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e706fa-e664-4779-ad96-16c45d83ebbe None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:33.930362 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e706fa-e664-4779-ad96-16c45d83ebbe None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:33.931254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e706fa-e664-4779-ad96-16c45d83ebbe None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:33.931761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e706fa-e664-4779-ad96-16c45d83ebbe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:33.933356 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e706fa-e664-4779-ad96-16c45d83ebbe None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:33.949135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 209/414] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:33.958324 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e706fa-e664-4779-ad96-16c45d83ebbe None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:34.077793 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bb0b85-69f0-40dc-8610-bca9f86ec33d None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:34.096420 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:34.123172 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97324cad-84a5-450a-b0be-33eeaf147aa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:34.124088 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97324cad-84a5-450a-b0be-33eeaf147aa3 None admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:34.124456 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97324cad-84a5-450a-b0be-33eeaf147aa3 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:34.124821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97324cad-84a5-450a-b0be-33eeaf147aa3 None admin] PATH_INFO: `/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/807b123652d245269336fcffaf36478e/roles/5a0f6f642a10411da6fc1bfc92a7a3d3` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:34.142315 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97324cad-84a5-450a-b0be-33eeaf147aa3 None admin] RBAC: Authorizing `identity:create_grant(project_id=658799ed31d2472fb36af4aff0f03aab, user_id=807b123652d245269336fcffaf36478e, role_id=5a0f6f642a10411da6fc1bfc92a7a3d3)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:34.144582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97324cad-84a5-450a-b0be-33eeaf147aa3 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:34.161309 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 206/415] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:43:33 2022] PUT /identity/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/807b123652d245269336fcffaf36478e/roles/5a0f6f642a10411da6fc1bfc92a7a3d3 => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:43:34.194185 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e706fa-e664-4779-ad96-16c45d83ebbe None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:34.248794 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433401e5-4095-4d82-9887-2c59d5940cf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:34.249629 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433401e5-4095-4d82-9887-2c59d5940cf8 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:34.249996 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433401e5-4095-4d82-9887-2c59d5940cf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:34.250357 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433401e5-4095-4d82-9887-2c59d5940cf8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:34.258852 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-433401e5-4095-4d82-9887-2c59d5940cf8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:34.260542 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-433401e5-4095-4d82-9887-2c59d5940cf8 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:34.273746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-433401e5-4095-4d82-9887-2c59d5940cf8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:34.273746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:34.273746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:34.273746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:34.273746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:34.273746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:34.273746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:34.273746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:34.273746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:34.273746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:34.273746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:34.273746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:34.273746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:43:34.273746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:43:34.273746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:43:34.274814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:43:34.274814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:34.274814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:34.274814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:34.274814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:34.274814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:34.274814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:34.274814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:34.274814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:34.274814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:34.274814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:34.274814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:34.274814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:34.274814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:34.274814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:34.274814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:34.274814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:34.275885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:43:34.275885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:43:34.275885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:43:34.275885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:43:34.275885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:43:34.275885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:43:34.275885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:34.275885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:34.277651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 210/416] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:43:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 320 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:34.283337 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97324cad-84a5-450a-b0be-33eeaf147aa3 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:34.394169 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97324cad-84a5-450a-b0be-33eeaf147aa3 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:34.429852 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ad3b02-ec2e-4bd7-9381-73ef83de5ba6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:34.430510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ad3b02-ec2e-4bd7-9381-73ef83de5ba6 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:34.430786 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ad3b02-ec2e-4bd7-9381-73ef83de5ba6 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:34.431050 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ad3b02-ec2e-4bd7-9381-73ef83de5ba6 None admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:34.431760 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ad3b02-ec2e-4bd7-9381-73ef83de5ba6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:34.432184 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ad3b02-ec2e-4bd7-9381-73ef83de5ba6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:34.433771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ad3b02-ec2e-4bd7-9381-73ef83de5ba6 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:34.446909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 207/417] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:43:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:34.452812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433401e5-4095-4d82-9887-2c59d5940cf8 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:34.474281 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:34.568514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433401e5-4095-4d82-9887-2c59d5940cf8 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:34.607742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e9bb4d-f960-4584-b5d6-95ca2c5e8bc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:34.608413 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e9bb4d-f960-4584-b5d6-95ca2c5e8bc1 None admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:34.608688 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e9bb4d-f960-4584-b5d6-95ca2c5e8bc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:34.608957 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e9bb4d-f960-4584-b5d6-95ca2c5e8bc1 None admin] PATH_INFO: `/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/81999329edfb4d4cac567d88d1f52909/roles/5a0f6f642a10411da6fc1bfc92a7a3d3` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:34.624413 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e9bb4d-f960-4584-b5d6-95ca2c5e8bc1 None admin] RBAC: Authorizing `identity:create_grant(project_id=658799ed31d2472fb36af4aff0f03aab, user_id=81999329edfb4d4cac567d88d1f52909, role_id=5a0f6f642a10411da6fc1bfc92a7a3d3)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:34.625843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e9bb4d-f960-4584-b5d6-95ca2c5e8bc1 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:34.648534 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 211/418] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:43:34 2022] PUT /identity/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/81999329edfb4d4cac567d88d1f52909/roles/5a0f6f642a10411da6fc1bfc92a7a3d3 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:43:35.137080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6158391-f493-4ec5-a1d4-97360f719b87 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:35.138117 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6158391-f493-4ec5-a1d4-97360f719b87 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:35.138601 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6158391-f493-4ec5-a1d4-97360f719b87 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:35.140427 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 208/419] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:35.147525 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c680a5-87e1-4d9e-b5b8-325946cc5f04 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:35.148074 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c680a5-87e1-4d9e-b5b8-325946cc5f04 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:35.148554 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c680a5-87e1-4d9e-b5b8-325946cc5f04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:35.172685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-69c680a5-87e1-4d9e-b5b8-325946cc5f04 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:35.174291 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-69c680a5-87e1-4d9e-b5b8-325946cc5f04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:35.254310 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:35.263160 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 212/420] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:35.471908 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93020ab8-93ac-4bef-ae85-a725f9d814f0 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:35.472529 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93020ab8-93ac-4bef-ae85-a725f9d814f0 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:35.472988 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93020ab8-93ac-4bef-ae85-a725f9d814f0 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:35.474721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 209/421] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:35.485116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d14323d-b3ff-43bb-9bab-fc07bea63324 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:35.485969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d14323d-b3ff-43bb-9bab-fc07bea63324 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:35.486436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d14323d-b3ff-43bb-9bab-fc07bea63324 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:35.516712 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-8d14323d-b3ff-43bb-9bab-fc07bea63324 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:35.518279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-8d14323d-b3ff-43bb-9bab-fc07bea63324 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:35.603808 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 213/422] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:35.638396 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93020ab8-93ac-4bef-ae85-a725f9d814f0 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:35.668636 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:35.757887 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93020ab8-93ac-4bef-ae85-a725f9d814f0 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:35.798819 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321e2eed-3f0d-4d8c-b9b7-6de68bd53358 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:35.799561 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321e2eed-3f0d-4d8c-b9b7-6de68bd53358 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:35.799866 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321e2eed-3f0d-4d8c-b9b7-6de68bd53358 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:35.800135 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321e2eed-3f0d-4d8c-b9b7-6de68bd53358 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:35.813338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321e2eed-3f0d-4d8c-b9b7-6de68bd53358 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:35.818771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-321e2eed-3f0d-4d8c-b9b7-6de68bd53358 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:35.818771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:35.818771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:35.818771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:35.818771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:35.818771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:35.818771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:35.818771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:35.818771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:35.818771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:35.818771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:35.818771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:35.818771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:35.818771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:35.818771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:35.818771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:35.819640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:35.819640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:35.819640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:35.819640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:35.819640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:35.819640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:35.819640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:35.819640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:35.819640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:35.819640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:35.819640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:35.819640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:35.819640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:35.819640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:35.819640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:35.819640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:35.820463 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:35.820463 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:35.820463 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:35.820463 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:35.820463 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:35.820463 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:35.820463 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:35.820463 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:35.820463 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:35.820463 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:35.820463 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:35.820463 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:35.824083 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 210/423] 10.222.0.16 () {56 vars in 1102 bytes} [Sun Oct 30 00:43:35 2022] GET /identity/v3/roles/service => generated 85 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:35.828839 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d14323d-b3ff-43bb-9bab-fc07bea63324 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:35.939024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321e2eed-3f0d-4d8c-b9b7-6de68bd53358 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:35.978164 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d14323d-b3ff-43bb-9bab-fc07bea63324 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:36.020206 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:36.041555 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c8554f-4bee-49e3-97b7-63c1188f1482 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:36.042225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c8554f-4bee-49e3-97b7-63c1188f1482 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:36.042498 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c8554f-4bee-49e3-97b7-63c1188f1482 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:36.042822 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c8554f-4bee-49e3-97b7-63c1188f1482 None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:36.043553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c8554f-4bee-49e3-97b7-63c1188f1482 None admin] RBAC: Adding query filter params (name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:36.043878 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c8554f-4bee-49e3-97b7-63c1188f1482 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:36.046735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c8554f-4bee-49e3-97b7-63c1188f1482 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:36.065827 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 214/424] 10.222.0.16 () {56 vars in 1103 bytes} [Sun Oct 30 00:43:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:36.066407 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:36.071123 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c8554f-4bee-49e3-97b7-63c1188f1482 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:36.085644 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321e2eed-3f0d-4d8c-b9b7-6de68bd53358 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:36.152957 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b559090b-fcb8-477a-8e4b-e5bf91a81ce9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:36.154404 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b559090b-fcb8-477a-8e4b-e5bf91a81ce9 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:36.154869 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b559090b-fcb8-477a-8e4b-e5bf91a81ce9 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:36.155336 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b559090b-fcb8-477a-8e4b-e5bf91a81ce9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:36.180235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b559090b-fcb8-477a-8e4b-e5bf91a81ce9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:36.194153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-b559090b-fcb8-477a-8e4b-e5bf91a81ce9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:36.194153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:36.194153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:36.194153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:36.194153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:36.194153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:36.194153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:36.194153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:36.194153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:36.194153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:36.194153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:36.194153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:36.194153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:36.194153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:36.194153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:36.194153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:36.195029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:36.195029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:36.195029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:36.195029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:36.195029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:36.195029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:36.195029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:36.195029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:36.195029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:36.195029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:36.195029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:36.195029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:36.195029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:36.195029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:36.195029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:36.195029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:36.195868 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:36.195868 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:36.195868 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:36.195868 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:36.195868 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:36.195868 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:36.195868 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:36.195868 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:36.195868 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:36.195868 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:36.195868 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:36.195868 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:36.195868 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 211/425] 10.222.0.16 () {56 vars in 1102 bytes} [Sun Oct 30 00:43:35 2022] GET /identity/v3/roles/service => generated 85 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:43:36.202690 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b559090b-fcb8-477a-8e4b-e5bf91a81ce9 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:36.237495 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c8554f-4bee-49e3-97b7-63c1188f1482 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:36.311320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b559090b-fcb8-477a-8e4b-e5bf91a81ce9 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:36.351738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68834cb7-9b0a-43ce-96d2-210e4588073c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:36.351738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68834cb7-9b0a-43ce-96d2-210e4588073c None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:36.351738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68834cb7-9b0a-43ce-96d2-210e4588073c None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:36.351738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68834cb7-9b0a-43ce-96d2-210e4588073c None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:36.352072 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68834cb7-9b0a-43ce-96d2-210e4588073c None admin] RBAC: Adding query filter params (name=service) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:36.352072 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68834cb7-9b0a-43ce-96d2-210e4588073c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:36.352072 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68834cb7-9b0a-43ce-96d2-210e4588073c None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:36.355304 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282d09a8-1e9c-46f1-ad82-7a07fc9ee90b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:36.355984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282d09a8-1e9c-46f1-ad82-7a07fc9ee90b None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:36.356262 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282d09a8-1e9c-46f1-ad82-7a07fc9ee90b None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:36.356531 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282d09a8-1e9c-46f1-ad82-7a07fc9ee90b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:36.359959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 212/426] 10.222.0.16 () {56 vars in 1103 bytes} [Sun Oct 30 00:43:36 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:36.370452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68834cb7-9b0a-43ce-96d2-210e4588073c None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:36.374249 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:36.387961 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282d09a8-1e9c-46f1-ad82-7a07fc9ee90b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:36.390581 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282d09a8-1e9c-46f1-ad82-7a07fc9ee90b None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:36.417930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-282d09a8-1e9c-46f1-ad82-7a07fc9ee90b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:36.417930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:36.417930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:36.417930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:36.417930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:36.417930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:36.417930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:36.417930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:36.417930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:36.417930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:36.417930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:36.417930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:36.417930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:36.417930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:36.417930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:36.417930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:36.418876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:36.418876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:36.418876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:36.418876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:36.418876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:36.418876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:36.418876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:36.418876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:36.418876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:36.418876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:36.418876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:36.418876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:36.418876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:36.418876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:36.418876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:36.418876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:36.418876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:36.419774 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:36.419774 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:36.419774 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:36.421778 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 215/427] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 351 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:43:36.427294 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282d09a8-1e9c-46f1-ad82-7a07fc9ee90b None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:36.512059 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:36.548885 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68834cb7-9b0a-43ce-96d2-210e4588073c None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:36.604988 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6370c7-5f0e-4250-9e2d-c106acbdba2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:36.605683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6370c7-5f0e-4250-9e2d-c106acbdba2f None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:36.606047 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6370c7-5f0e-4250-9e2d-c106acbdba2f None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:36.606402 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6370c7-5f0e-4250-9e2d-c106acbdba2f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:36.620907 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6370c7-5f0e-4250-9e2d-c106acbdba2f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:36.622648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6370c7-5f0e-4250-9e2d-c106acbdba2f None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:36.637882 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282d09a8-1e9c-46f1-ad82-7a07fc9ee90b None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:36.639164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-2b6370c7-5f0e-4250-9e2d-c106acbdba2f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:36.639164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:36.639164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:36.639164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:36.639164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:36.639164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:36.639164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:36.639164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:36.639164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:36.639164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:36.639164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:36.639164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:36.639164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:36.639164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:36.639164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:36.639164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:36.640234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:36.640234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:36.640234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:36.640234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:36.640234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:36.640234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:36.640234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:36.640234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:36.640234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:36.640234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:36.640234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:36.640234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:36.640234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:36.640234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:36.640234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:36.640234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:36.640234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:36.641099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:36.641099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:36.641099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:36.641553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 213/428] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:36.647004 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6370c7-5f0e-4250-9e2d-c106acbdba2f None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:36.697941 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0d482b-7bd2-4d57-97f6-e402ccb47c0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:36.698546 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0d482b-7bd2-4d57-97f6-e402ccb47c0a None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:36.698816 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0d482b-7bd2-4d57-97f6-e402ccb47c0a None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:36.699080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0d482b-7bd2-4d57-97f6-e402ccb47c0a None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:36.699742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0d482b-7bd2-4d57-97f6-e402ccb47c0a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:36.700032 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0d482b-7bd2-4d57-97f6-e402ccb47c0a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:36.701398 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0d482b-7bd2-4d57-97f6-e402ccb47c0a None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:36.728633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 216/429] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:36.729899 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad04d2c3-9f8f-4069-ab1c-b98b1349b756 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:36.730173 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad04d2c3-9f8f-4069-ab1c-b98b1349b756 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:36.730433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad04d2c3-9f8f-4069-ab1c-b98b1349b756 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:36.731937 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 217/430] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:36.737553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad04d2c3-9f8f-4069-ab1c-b98b1349b756 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:36.747209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:36.799970 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6370c7-5f0e-4250-9e2d-c106acbdba2f None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:36.847960 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a9cae1-f7bd-43ef-a648-441aa21c1cf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:36.848823 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a9cae1-f7bd-43ef-a648-441aa21c1cf7 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:36.849193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a9cae1-f7bd-43ef-a648-441aa21c1cf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:36.849625 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a9cae1-f7bd-43ef-a648-441aa21c1cf7 None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:36.850682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a9cae1-f7bd-43ef-a648-441aa21c1cf7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:36.850820 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a9cae1-f7bd-43ef-a648-441aa21c1cf7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:36.852294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a9cae1-f7bd-43ef-a648-441aa21c1cf7 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:36.854401 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:36.861051 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad04d2c3-9f8f-4069-ab1c-b98b1349b756 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:36.873219 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 214/431] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:36.874898 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a42879c-c694-480c-a9f3-a37688bb3f21 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:36.875286 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a42879c-c694-480c-a9f3-a37688bb3f21 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:36.875646 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a42879c-c694-480c-a9f3-a37688bb3f21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:36.901120 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b720175d-7a5a-4db6-ae73-9ab11765a9ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:36.902002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b720175d-7a5a-4db6-ae73-9ab11765a9ef None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:36.902448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b720175d-7a5a-4db6-ae73-9ab11765a9ef None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:36.902853 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b720175d-7a5a-4db6-ae73-9ab11765a9ef None admin] PATH_INFO: `/v3/users/swift` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:36.903857 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-4a42879c-c694-480c-a9f3-a37688bb3f21 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:36.905453 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-4a42879c-c694-480c-a9f3-a37688bb3f21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:36.912315 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b720175d-7a5a-4db6-ae73-9ab11765a9ef None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:36.913978 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b720175d-7a5a-4db6-ae73-9ab11765a9ef None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:36.930520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-b720175d-7a5a-4db6-ae73-9ab11765a9ef None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 30 00:43:36.930520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:36.930520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:36.930520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:36.930520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:36.930520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:36.930520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:36.930520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:36.930520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:36.930520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:36.930520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:36.930520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:36.930520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:43:36.930520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:43:36.930520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:43:36.930520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:43:36.931486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:36.931486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:36.931486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:43:36.931486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:36.931486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:43:36.931486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:36.931486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:36.931486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:36.931486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:36.931486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:36.931486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:36.931486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:36.931486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:36.931486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:36.931486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:36.931486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:36.931486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:36.931486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:36.932442 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:36.932442 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:36.932442 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:43:36.932442 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:43:36.932442 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:43:36.932442 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:43:36.932442 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:43:36.932442 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:43:36.932442 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 30 00:43:36.932442 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:36.938388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 218/432] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:43:36 2022] GET /identity/v3/users/swift => generated 83 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:36.938388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b720175d-7a5a-4db6-ae73-9ab11765a9ef None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:36.989061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 215/433] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:36.990245 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a42879c-c694-480c-a9f3-a37688bb3f21 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:37.050986 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:37.062276 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b720175d-7a5a-4db6-ae73-9ab11765a9ef None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:37.107221 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a42879c-c694-480c-a9f3-a37688bb3f21 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:37.112285 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8311cb-f858-4deb-be79-7d6a5d54b2ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:37.113263 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8311cb-f858-4deb-be79-7d6a5d54b2ba None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:37.113731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8311cb-f858-4deb-be79-7d6a5d54b2ba None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:37.114259 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8311cb-f858-4deb-be79-7d6a5d54b2ba None admin] PATH_INFO: `/v3/users/nova` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:37.127213 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8311cb-f858-4deb-be79-7d6a5d54b2ba None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:37.129843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8311cb-f858-4deb-be79-7d6a5d54b2ba None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:37.143017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0efb4ca4-2f92-4b09-b2d9-74278c538fe1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:37.143643 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efb4ca4-2f92-4b09-b2d9-74278c538fe1 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:37.143958 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efb4ca4-2f92-4b09-b2d9-74278c538fe1 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:37.144224 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efb4ca4-2f92-4b09-b2d9-74278c538fe1 None admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:37.144946 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0efb4ca4-2f92-4b09-b2d9-74278c538fe1 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:37.145252 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0efb4ca4-2f92-4b09-b2d9-74278c538fe1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:37.146702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0efb4ca4-2f92-4b09-b2d9-74278c538fe1 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:37.147588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-ab8311cb-f858-4deb-be79-7d6a5d54b2ba None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 30 00:43:37.147588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:37.147588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:37.147588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:37.147588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:37.147588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:37.147588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:37.147588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:37.147588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:37.147588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:37.147588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:37.147588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:37.147588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:43:37.147588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:43:37.147588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:43:37.147588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:43:37.148503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:37.148503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:37.148503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:43:37.148503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:37.148503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:43:37.148503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:37.148503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:37.148503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:37.148503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:37.148503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:37.148503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:37.148503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:37.148503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:37.148503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:37.148503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:37.148503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:37.148503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:37.148503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:37.149495 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:37.149495 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:37.149495 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:43:37.149495 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:43:37.149495 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:43:37.149495 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:43:37.149495 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:43:37.149495 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:43:37.149495 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 30 00:43:37.149495 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:37.153248 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 219/434] 10.222.0.16 () {56 vars in 1093 bytes} [Sun Oct 30 00:43:36 2022] GET /identity/v3/users/nova => generated 82 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:37.154774 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d857bbc-d24a-4fa4-ad91-7157890e6f6f None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:37.155242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d857bbc-d24a-4fa4-ad91-7157890e6f6f None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:37.155979 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d857bbc-d24a-4fa4-ad91-7157890e6f6f None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:37.157954 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 220/435] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:37.159023 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d857bbc-d24a-4fa4-ad91-7157890e6f6f None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:37.173817 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 216/436] 10.222.0.16 () {56 vars in 1135 bytes} [Sun Oct 30 00:43:36 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:37.179066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a312870-ad84-4620-a6d7-0a169d2f1eeb None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:37.179066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a312870-ad84-4620-a6d7-0a169d2f1eeb None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:37.179066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a312870-ad84-4620-a6d7-0a169d2f1eeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:37.199058 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-3a312870-ad84-4620-a6d7-0a169d2f1eeb None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:37.200323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3a312870-ad84-4620-a6d7-0a169d2f1eeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:37.208143 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:37.283014 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 217/437] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:37.283976 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a312870-ad84-4620-a6d7-0a169d2f1eeb None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:37.318585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d857bbc-d24a-4fa4-ad91-7157890e6f6f None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:37.320592 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:37.361296 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f544679f-8edb-4665-a773-41e512f7df9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:37.364919 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f544679f-8edb-4665-a773-41e512f7df9d None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:37.365419 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f544679f-8edb-4665-a773-41e512f7df9d None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:37.365903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f544679f-8edb-4665-a773-41e512f7df9d None admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:37.367188 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f544679f-8edb-4665-a773-41e512f7df9d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:37.367486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f544679f-8edb-4665-a773-41e512f7df9d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:37.369275 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f544679f-8edb-4665-a773-41e512f7df9d None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:37.386483 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 221/438] 10.222.0.16 () {56 vars in 1133 bytes} [Sun Oct 30 00:43:37 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:37.386483 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f544679f-8edb-4665-a773-41e512f7df9d None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:37.396668 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a312870-ad84-4620-a6d7-0a169d2f1eeb None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:37.433132 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797fb344-f5af-4428-b643-b9df2c3584bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:37.433759 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797fb344-f5af-4428-b643-b9df2c3584bb None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:37.434080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797fb344-f5af-4428-b643-b9df2c3584bb None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:37.434299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797fb344-f5af-4428-b643-b9df2c3584bb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:37.444872 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797fb344-f5af-4428-b643-b9df2c3584bb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:37.446338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797fb344-f5af-4428-b643-b9df2c3584bb None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:37.461060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-797fb344-f5af-4428-b643-b9df2c3584bb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:37.461060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:37.461060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:37.461060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:37.461060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:37.461060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:37.461060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:37.461060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:37.461060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:37.461060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:37.461060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:37.461060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:37.461060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:37.461060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:37.461060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:37.461060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:37.461967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:37.461967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:37.461967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:37.461967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:37.461967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:37.461967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:37.461967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:37.461967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:37.461967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:37.461967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:37.461967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:37.461967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:37.461967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:37.461967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:37.461967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:37.461967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:37.461967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:37.462844 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:37.462844 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:37.462844 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:37.462844 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 218/439] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:37.464155 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4105008f-0c29-4686-a8e3-ce2f2238fc18 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:37.464155 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4105008f-0c29-4686-a8e3-ce2f2238fc18 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:37.464155 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4105008f-0c29-4686-a8e3-ce2f2238fc18 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:37.465156 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 219/440] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:37 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:37.466000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4105008f-0c29-4686-a8e3-ce2f2238fc18 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:37.474630 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:37.497861 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f544679f-8edb-4665-a773-41e512f7df9d None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:37.533007 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7abd42a-8067-4f6e-be5e-fb363f70b7c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:37.533809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7abd42a-8067-4f6e-be5e-fb363f70b7c1 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:37.534135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7abd42a-8067-4f6e-be5e-fb363f70b7c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:37.534396 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7abd42a-8067-4f6e-be5e-fb363f70b7c1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:37.545779 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7abd42a-8067-4f6e-be5e-fb363f70b7c1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:37.551797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-e7abd42a-8067-4f6e-be5e-fb363f70b7c1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:37.551797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:37.551797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:37.551797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:37.551797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:37.551797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:37.551797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:37.551797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:37.551797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:37.551797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:37.551797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:37.551797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:37.551797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:37.551797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:37.551797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:37.551797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:37.552704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:37.552704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:37.552704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:37.552704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:37.552704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:37.552704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:37.552704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:37.552704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:37.552704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:37.552704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:37.552704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:37.552704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:37.552704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:37.552704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:37.552704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:37.552704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:37.554378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:37.554378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:37.554378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:37.554378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:37.554378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:37.554378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:37.554378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:37.554378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:37.554378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:37.554378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:37.554378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:37.554378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:37.554378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 222/441] 10.222.0.16 () {56 vars in 1102 bytes} [Sun Oct 30 00:43:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:37.554378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7abd42a-8067-4f6e-be5e-fb363f70b7c1 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:37.558832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:37.576130 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4105008f-0c29-4686-a8e3-ce2f2238fc18 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:37.608301 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d3e3b8-07d5-464b-8f20-b353e07ccf57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:37.608902 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d3e3b8-07d5-464b-8f20-b353e07ccf57 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:37.609177 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d3e3b8-07d5-464b-8f20-b353e07ccf57 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:37.609444 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d3e3b8-07d5-464b-8f20-b353e07ccf57 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:37.649953 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7abd42a-8067-4f6e-be5e-fb363f70b7c1 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:37.675890 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d3e3b8-07d5-464b-8f20-b353e07ccf57 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:37.677379 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d3e3b8-07d5-464b-8f20-b353e07ccf57 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:37.682212 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bafc474-3dc2-4f6d-8e36-ee259a290382 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:37.682809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bafc474-3dc2-4f6d-8e36-ee259a290382 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:37.683082 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bafc474-3dc2-4f6d-8e36-ee259a290382 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:37.683350 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bafc474-3dc2-4f6d-8e36-ee259a290382 None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:37.684014 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bafc474-3dc2-4f6d-8e36-ee259a290382 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:37.684312 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bafc474-3dc2-4f6d-8e36-ee259a290382 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:37.685986 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bafc474-3dc2-4f6d-8e36-ee259a290382 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:37.692484 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:37.697973 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 223/442] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:37.699221 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f01048-3188-4407-b8a1-61822436269d None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:37.699500 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f01048-3188-4407-b8a1-61822436269d None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:37.699800 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f01048-3188-4407-b8a1-61822436269d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:37.717794 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-96f01048-3188-4407-b8a1-61822436269d None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:37.718985 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-96f01048-3188-4407-b8a1-61822436269d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:37.763899 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:37.779825 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 220/443] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:43:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 30 00:43:37.780638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d3e3b8-07d5-464b-8f20-b353e07ccf57 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:37.784986 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 224/444] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:37.785875 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f01048-3188-4407-b8a1-61822436269d None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:37.887973 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d3e3b8-07d5-464b-8f20-b353e07ccf57 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:37.888334 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f01048-3188-4407-b8a1-61822436269d None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:37.916019 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:37.924848 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff257a10-56ac-41d6-a4d6-d0b71baf6e66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:37.925457 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff257a10-56ac-41d6-a4d6-d0b71baf6e66 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:37.925733 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff257a10-56ac-41d6-a4d6-d0b71baf6e66 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:37.925998 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff257a10-56ac-41d6-a4d6-d0b71baf6e66 None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:37.926254 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8371dbd4-e6ab-4fb9-9a19-36284e31fbe4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:37.926653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff257a10-56ac-41d6-a4d6-d0b71baf6e66 None admin] RBAC: Adding query filter params (name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:37.926878 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8371dbd4-e6ab-4fb9-9a19-36284e31fbe4 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:37.926968 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff257a10-56ac-41d6-a4d6-d0b71baf6e66 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:37.927182 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8371dbd4-e6ab-4fb9-9a19-36284e31fbe4 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:37.927447 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8371dbd4-e6ab-4fb9-9a19-36284e31fbe4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:37.928375 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff257a10-56ac-41d6-a4d6-d0b71baf6e66 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:37.937117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 225/445] 10.222.0.16 () {56 vars in 1103 bytes} [Sun Oct 30 00:43:37 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:37.938131 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff257a10-56ac-41d6-a4d6-d0b71baf6e66 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:37.943011 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8371dbd4-e6ab-4fb9-9a19-36284e31fbe4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:37.947982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-8371dbd4-e6ab-4fb9-9a19-36284e31fbe4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:37.947982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:37.947982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:37.947982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:37.947982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:37.947982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:37.947982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:37.947982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:37.947982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:37.947982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:37.947982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:37.947982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:37.947982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:37.947982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:37.947982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:37.947982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:37.948861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:37.948861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:37.948861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:37.948861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:37.948861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:37.948861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:37.948861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:37.948861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:37.948861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:37.948861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:37.948861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:37.948861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:37.948861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:37.948861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:37.948861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:37.948861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:37.949660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:37.949660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:37.949660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:37.949660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:37.949660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:37.949660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:37.949660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:37.949660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:37.949660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:37.949660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:37.949660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:37.949660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:37.949660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 221/446] 10.222.0.16 () {56 vars in 1102 bytes} [Sun Oct 30 00:43:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:37.950302 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8371dbd4-e6ab-4fb9-9a19-36284e31fbe4 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:37.999960 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:38.043151 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff257a10-56ac-41d6-a4d6-d0b71baf6e66 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:38.054362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8371dbd4-e6ab-4fb9-9a19-36284e31fbe4 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:38.076250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf31dc25-b66c-4a07-b713-668851cdabe5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:38.076866 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf31dc25-b66c-4a07-b713-668851cdabe5 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:38.077136 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf31dc25-b66c-4a07-b713-668851cdabe5 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:38.077416 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf31dc25-b66c-4a07-b713-668851cdabe5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:38.084003 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf31dc25-b66c-4a07-b713-668851cdabe5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:38.085430 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf31dc25-b66c-4a07-b713-668851cdabe5 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:38.087253 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc97729e-9306-4733-a366-261f4041b903 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:38.087894 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc97729e-9306-4733-a366-261f4041b903 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:38.088164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc97729e-9306-4733-a366-261f4041b903 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:38.088427 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc97729e-9306-4733-a366-261f4041b903 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:38.096209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-bf31dc25-b66c-4a07-b713-668851cdabe5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:38.096209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:38.096209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:38.096209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:38.096209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:38.096209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:38.096209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:38.096209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:38.096209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:38.096209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:38.096209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:38.096209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:38.096209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:43:38.096209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:43:38.096209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:43:38.097050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:43:38.097050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:38.097050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:38.097050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:38.097050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:38.097050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:38.097050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:38.097050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:38.097050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:38.097050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:38.097050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:38.097050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:38.097050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:38.097050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:38.097050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:38.097050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:38.097050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:38.097927 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:43:38.097927 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:43:38.097927 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:43:38.097927 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:43:38.097927 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:43:38.097927 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:43:38.097927 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:38.097927 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:38.097927 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 226/447] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:43:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:38.098380 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc97729e-9306-4733-a366-261f4041b903 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:38.098447 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf31dc25-b66c-4a07-b713-668851cdabe5 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:38.099442 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc97729e-9306-4733-a366-261f4041b903 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:38.117881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-cc97729e-9306-4733-a366-261f4041b903 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:38.117881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:38.117881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:38.117881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:38.117881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:38.117881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:38.117881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:38.117881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:38.117881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:38.117881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:38.117881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:38.117881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:38.117881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:38.117881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:38.117881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:38.117881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:38.123428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:38.123428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:38.123428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:38.123428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:38.123428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:38.123428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:38.123428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:38.123428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:38.123428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:38.123428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:38.123428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:38.123428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:38.123428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:38.123428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:38.123428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:38.123428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:38.123428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:38.124556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:38.124556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:38.124556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:38.124556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 222/448] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:38.124556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc97729e-9306-4733-a366-261f4041b903 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:38.156032 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:38.207138 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf31dc25-b66c-4a07-b713-668851cdabe5 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:38.214408 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:38.218230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc97729e-9306-4733-a366-261f4041b903 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:38.239853 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29f8039-db5f-4764-b9ba-78929e0ea944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:38.240482 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29f8039-db5f-4764-b9ba-78929e0ea944 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:38.240755 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29f8039-db5f-4764-b9ba-78929e0ea944 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:38.241024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29f8039-db5f-4764-b9ba-78929e0ea944 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:38.250949 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4de035-4ca5-45c8-b4d6-ab49deaa6741 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:38.251387 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29f8039-db5f-4764-b9ba-78929e0ea944 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:38.251545 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4de035-4ca5-45c8-b4d6-ab49deaa6741 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:38.251933 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4de035-4ca5-45c8-b4d6-ab49deaa6741 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:38.252545 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4de035-4ca5-45c8-b4d6-ab49deaa6741 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:38.252913 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29f8039-db5f-4764-b9ba-78929e0ea944 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:38.253194 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4de035-4ca5-45c8-b4d6-ab49deaa6741 None admin] RBAC: Adding query filter params (name=service) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:38.253497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4de035-4ca5-45c8-b4d6-ab49deaa6741 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:38.254859 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4de035-4ca5-45c8-b4d6-ab49deaa6741 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:38.262791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 223/449] 10.222.0.16 () {56 vars in 1103 bytes} [Sun Oct 30 00:43:38 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:38.263578 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4de035-4ca5-45c8-b4d6-ab49deaa6741 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:38.266514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-f29f8039-db5f-4764-b9ba-78929e0ea944 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:38.266514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:38.266514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:38.266514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:38.266514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:38.266514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:38.266514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:38.266514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:38.266514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:38.266514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:38.266514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:38.266514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:38.266514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:38.266514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:38.266514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:38.266514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:38.267380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:38.267380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:38.267380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:38.267380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:38.267380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:38.267380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:38.267380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:38.267380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:38.267380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:38.267380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:38.267380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:38.267380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:38.267380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:38.267380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:38.267380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:38.267380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:38.267380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:38.268529 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:38.268529 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:38.268529 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:38.268529 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 227/450] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:38.268847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29f8039-db5f-4764-b9ba-78929e0ea944 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:38.369743 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29f8039-db5f-4764-b9ba-78929e0ea944 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:38.370334 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4de035-4ca5-45c8-b4d6-ab49deaa6741 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:38.371258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:38.402745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7896adc-77fb-4331-912b-817172bdcff0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:38.403339 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7896adc-77fb-4331-912b-817172bdcff0 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:38.403606 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7896adc-77fb-4331-912b-817172bdcff0 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:38.403903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7896adc-77fb-4331-912b-817172bdcff0 None admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:38.404582 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7896adc-77fb-4331-912b-817172bdcff0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:38.404876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7896adc-77fb-4331-912b-817172bdcff0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:38.405081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57194f40-b17c-401a-8917-0454190fa0b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:38.405689 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57194f40-b17c-401a-8917-0454190fa0b3 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:38.405961 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57194f40-b17c-401a-8917-0454190fa0b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:38.406225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57194f40-b17c-401a-8917-0454190fa0b3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:38.406321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7896adc-77fb-4331-912b-817172bdcff0 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:38.416471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57194f40-b17c-401a-8917-0454190fa0b3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:38.418077 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 228/451] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:43:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:38.418935 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7896adc-77fb-4331-912b-817172bdcff0 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:38.421629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-57194f40-b17c-401a-8917-0454190fa0b3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:38.421629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:38.421629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:38.421629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:38.421629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:38.421629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:38.421629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:38.421629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:38.421629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:38.421629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:38.421629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:38.421629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:38.421629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:38.421629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:38.421629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:38.421629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:38.422657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:38.422657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:38.422657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:38.422657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:38.422657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:38.422657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:38.422657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:38.422657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:38.422657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:38.422657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:38.422657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:38.422657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:38.422657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:38.422657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:38.422657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:38.422657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:38.423633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:38.423633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:38.423633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:38.423633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:38.423633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:38.423633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:38.423633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:38.423633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:38.423633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:38.423633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:38.423633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:38.423633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:38.423633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 224/452] 10.222.0.16 () {56 vars in 1102 bytes} [Sun Oct 30 00:43:38 2022] GET /identity/v3/roles/service => generated 85 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:38.424426 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57194f40-b17c-401a-8917-0454190fa0b3 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:38.446915 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:38.522773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7896adc-77fb-4331-912b-817172bdcff0 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:38.523915 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57194f40-b17c-401a-8917-0454190fa0b3 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:38.555783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03eb47f9-d793-4a2c-ade9-d1f5e60c240f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:38.556315 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e75a1d-e640-44a7-9b71-d50336610e30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:38.556420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eb47f9-d793-4a2c-ade9-d1f5e60c240f None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:38.556643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eb47f9-d793-4a2c-ade9-d1f5e60c240f None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:38.556742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e75a1d-e640-44a7-9b71-d50336610e30 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:38.556905 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eb47f9-d793-4a2c-ade9-d1f5e60c240f None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:38.557002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e75a1d-e640-44a7-9b71-d50336610e30 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:38.557264 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e75a1d-e640-44a7-9b71-d50336610e30 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:38.557532 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03eb47f9-d793-4a2c-ade9-d1f5e60c240f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:38.557831 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03eb47f9-d793-4a2c-ade9-d1f5e60c240f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:38.559207 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03eb47f9-d793-4a2c-ade9-d1f5e60c240f None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:38.569337 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e75a1d-e640-44a7-9b71-d50336610e30 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:38.570681 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 229/453] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:38.570817 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e75a1d-e640-44a7-9b71-d50336610e30 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:38.571471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03eb47f9-d793-4a2c-ade9-d1f5e60c240f None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:38.584662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-c4e75a1d-e640-44a7-9b71-d50336610e30 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:38.584662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:38.584662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:38.584662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:38.584662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:38.584662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:38.584662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:38.584662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:38.584662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:38.584662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:38.584662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:38.584662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:38.584662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:38.584662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:38.584662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:38.584662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:38.585804 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:38.585804 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:38.585804 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:38.585804 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:38.585804 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:38.585804 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:38.585804 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:38.585804 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:38.585804 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:38.585804 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:38.585804 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:38.585804 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:38.585804 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:38.585804 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:38.585804 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:38.585804 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:38.585804 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:38.586849 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:38.586849 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:38.586849 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:38.588002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 225/454] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:38.588775 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e75a1d-e640-44a7-9b71-d50336610e30 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:38.599556 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:38.658465 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:38.674418 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03eb47f9-d793-4a2c-ade9-d1f5e60c240f None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:38.689174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e75a1d-e640-44a7-9b71-d50336610e30 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:38.706933 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df69da2b-0691-4f35-a304-0597592a33ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:38.707525 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df69da2b-0691-4f35-a304-0597592a33ad None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:38.707830 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df69da2b-0691-4f35-a304-0597592a33ad None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:38.708126 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df69da2b-0691-4f35-a304-0597592a33ad None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:38.708751 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df69da2b-0691-4f35-a304-0597592a33ad None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:38.709038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df69da2b-0691-4f35-a304-0597592a33ad None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:38.710384 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df69da2b-0691-4f35-a304-0597592a33ad None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:38.722066 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 230/455] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:38.726613 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df69da2b-0691-4f35-a304-0597592a33ad None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:38.728209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e30603-3817-45e3-852c-6eb839046c13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:38.728871 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e30603-3817-45e3-852c-6eb839046c13 None admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:38.729145 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e30603-3817-45e3-852c-6eb839046c13 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:38.729407 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e30603-3817-45e3-852c-6eb839046c13 None admin] PATH_INFO: `/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/67f6eff8c0254cc6a210101dd7af988d/roles/5a0f6f642a10411da6fc1bfc92a7a3d3` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:38.744193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e30603-3817-45e3-852c-6eb839046c13 None admin] RBAC: Authorizing `identity:create_grant(project_id=658799ed31d2472fb36af4aff0f03aab, user_id=67f6eff8c0254cc6a210101dd7af988d, role_id=5a0f6f642a10411da6fc1bfc92a7a3d3)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:38.745660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e30603-3817-45e3-852c-6eb839046c13 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:38.758600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 226/456] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:43:38 2022] PUT /identity/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/67f6eff8c0254cc6a210101dd7af988d/roles/5a0f6f642a10411da6fc1bfc92a7a3d3 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:43:38.759377 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e30603-3817-45e3-852c-6eb839046c13 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:38.821516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:38.842423 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df69da2b-0691-4f35-a304-0597592a33ad None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:38.876951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e30603-3817-45e3-852c-6eb839046c13 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:38.881123 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af617f34-e863-4e57-9da4-d956a35a7cd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:38.881818 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af617f34-e863-4e57-9da4-d956a35a7cd4 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:38.882152 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af617f34-e863-4e57-9da4-d956a35a7cd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:38.882478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af617f34-e863-4e57-9da4-d956a35a7cd4 None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:38.883202 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af617f34-e863-4e57-9da4-d956a35a7cd4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:38.883559 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af617f34-e863-4e57-9da4-d956a35a7cd4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:38.885044 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af617f34-e863-4e57-9da4-d956a35a7cd4 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:38.899749 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 231/457] 10.222.0.16 () {56 vars in 1103 bytes} [Sun Oct 30 00:43:38 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:38.899749 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af617f34-e863-4e57-9da4-d956a35a7cd4 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:38.908327 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:38.923553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658988a2-59e8-4bf1-a19c-f15481399918 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:38.924411 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658988a2-59e8-4bf1-a19c-f15481399918 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:38.924738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658988a2-59e8-4bf1-a19c-f15481399918 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:38.925106 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658988a2-59e8-4bf1-a19c-f15481399918 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:38.932495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658988a2-59e8-4bf1-a19c-f15481399918 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:38.934135 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658988a2-59e8-4bf1-a19c-f15481399918 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:38.945774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-658988a2-59e8-4bf1-a19c-f15481399918 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:38.945774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:38.945774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:38.945774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:38.945774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:38.945774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:38.945774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:38.945774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:38.945774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:38.945774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:38.945774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:38.945774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:38.945774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:43:38.945774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:43:38.945774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:43:38.946713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:43:38.946713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:38.946713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:38.946713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:38.946713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:38.946713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:38.946713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:38.946713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:38.946713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:38.946713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:38.946713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:38.946713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:38.946713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:38.946713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:38.946713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:38.946713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:38.946713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:38.947611 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:43:38.947611 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:43:38.947611 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:43:38.947611 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:43:38.947611 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:43:38.947611 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:43:38.947611 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:38.947611 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:38.948070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 227/458] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:43:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:38.948638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658988a2-59e8-4bf1-a19c-f15481399918 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:39.004033 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af617f34-e863-4e57-9da4-d956a35a7cd4 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:39.036284 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9434555-7eda-4c7e-bbb8-520f2529911a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:39.036886 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9434555-7eda-4c7e-bbb8-520f2529911a None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:39.037156 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9434555-7eda-4c7e-bbb8-520f2529911a None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:39.037421 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9434555-7eda-4c7e-bbb8-520f2529911a None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:39.038052 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9434555-7eda-4c7e-bbb8-520f2529911a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:39.038352 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9434555-7eda-4c7e-bbb8-520f2529911a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:39.039701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9434555-7eda-4c7e-bbb8-520f2529911a None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:39.051386 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 232/459] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:39.052218 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9434555-7eda-4c7e-bbb8-520f2529911a None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:39.057028 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:39.060303 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658988a2-59e8-4bf1-a19c-f15481399918 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:39.098051 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38102e87-c805-4a87-9ea9-a6217f3706d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:39.099028 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38102e87-c805-4a87-9ea9-a6217f3706d8 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:39.099438 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38102e87-c805-4a87-9ea9-a6217f3706d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:39.099888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38102e87-c805-4a87-9ea9-a6217f3706d8 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:39.107793 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38102e87-c805-4a87-9ea9-a6217f3706d8 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:39.109559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38102e87-c805-4a87-9ea9-a6217f3706d8 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:39.122073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-38102e87-c805-4a87-9ea9-a6217f3706d8 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 30 00:43:39.122073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:39.122073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:39.122073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:39.122073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:39.122073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:39.122073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:39.122073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:39.122073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:39.122073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:39.122073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:39.122073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:39.122073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:43:39.122073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:43:39.122073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:43:39.122073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:43:39.123160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:39.123160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:39.123160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:43:39.123160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:39.123160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:43:39.123160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:39.123160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:39.123160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:39.123160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:39.123160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:39.123160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:39.123160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:39.123160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:39.123160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:39.123160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:39.123160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:39.123160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:39.123160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:39.124180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:39.124180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:39.124180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:43:39.124180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:43:39.124180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:43:39.124180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:43:39.124180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:43:39.124180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:43:39.124180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 30 00:43:39.124180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:39.125134 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 228/460] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:43:38 2022] GET /identity/v3/users/cinder => generated 84 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:39.125912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38102e87-c805-4a87-9ea9-a6217f3706d8 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:39.154105 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9434555-7eda-4c7e-bbb8-520f2529911a None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:39.173203 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:39.194311 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bced33a-34a7-4dd3-884e-df99d316c28f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:39.195078 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bced33a-34a7-4dd3-884e-df99d316c28f None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:39.195430 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bced33a-34a7-4dd3-884e-df99d316c28f None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:39.195783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bced33a-34a7-4dd3-884e-df99d316c28f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:39.205952 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bced33a-34a7-4dd3-884e-df99d316c28f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:39.207475 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bced33a-34a7-4dd3-884e-df99d316c28f None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:39.222479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-4bced33a-34a7-4dd3-884e-df99d316c28f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:39.222479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:39.222479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:39.222479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:39.222479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:39.222479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:39.222479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:39.222479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:39.222479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:39.222479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:39.222479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:39.222479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:39.222479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:39.222479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:39.222479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:39.222479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:39.223610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:39.223610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:39.223610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:39.223610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:39.223610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:39.223610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:39.223610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:39.223610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:39.223610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:39.223610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:39.223610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:39.223610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:39.223610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:39.223610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:39.223610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:39.223610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:39.223610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:39.224567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:39.224567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:39.224567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:39.224567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 233/461] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:39.225058 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bced33a-34a7-4dd3-884e-df99d316c28f None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:39.241016 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38102e87-c805-4a87-9ea9-a6217f3706d8 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:39.277566 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cc89cc-4c59-4cda-9994-828b9363b14a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:39.278173 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cc89cc-4c59-4cda-9994-828b9363b14a None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:39.278449 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cc89cc-4c59-4cda-9994-828b9363b14a None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:39.278713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cc89cc-4c59-4cda-9994-828b9363b14a None admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:39.279406 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47cc89cc-4c59-4cda-9994-828b9363b14a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:39.279750 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47cc89cc-4c59-4cda-9994-828b9363b14a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:39.281606 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47cc89cc-4c59-4cda-9994-828b9363b14a None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:39.285657 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:39.294613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 229/462] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:43:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:39.295701 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cc89cc-4c59-4cda-9994-828b9363b14a None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:39.336769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bced33a-34a7-4dd3-884e-df99d316c28f None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:39.376056 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377a0298-78ee-4604-a6c8-347a1aab2a69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:39.376767 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377a0298-78ee-4604-a6c8-347a1aab2a69 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:39.377130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377a0298-78ee-4604-a6c8-347a1aab2a69 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:39.377611 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377a0298-78ee-4604-a6c8-347a1aab2a69 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:39.385827 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377a0298-78ee-4604-a6c8-347a1aab2a69 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:39.387578 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377a0298-78ee-4604-a6c8-347a1aab2a69 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:39.399177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-377a0298-78ee-4604-a6c8-347a1aab2a69 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 30 00:43:39.399177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:39.399177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:39.399177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:39.399177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:39.399177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:39.399177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:39.399177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:39.399177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:39.399177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:39.399177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:39.399177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:39.399177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:43:39.399177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:43:39.399177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:43:39.399177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:43:39.400327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:39.400327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:39.400327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:43:39.400327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:39.400327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:43:39.400327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:39.400327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:39.400327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:39.400327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:39.400327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:39.400327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:39.400327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:39.400327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:39.400327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:39.400327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:39.400327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:39.400327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:39.400327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:39.405883 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:39.405883 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:39.405883 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:43:39.405883 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:43:39.405883 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:43:39.405883 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:43:39.405883 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:43:39.405883 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:43:39.405883 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 30 00:43:39.405883 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:39.405883 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 234/463] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:43:39 2022] GET /identity/v3/users/glance => generated 84 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:39.405883 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377a0298-78ee-4604-a6c8-347a1aab2a69 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:39.409396 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cc89cc-4c59-4cda-9994-828b9363b14a None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:39.411012 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:39.451505 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ed0c9b-68a4-4095-ae36-4ca530d20c22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:39.452135 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ed0c9b-68a4-4095-ae36-4ca530d20c22 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:39.452408 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ed0c9b-68a4-4095-ae36-4ca530d20c22 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:39.452674 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ed0c9b-68a4-4095-ae36-4ca530d20c22 None admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:39.453408 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ed0c9b-68a4-4095-ae36-4ca530d20c22 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:39.453697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ed0c9b-68a4-4095-ae36-4ca530d20c22 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:39.455136 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ed0c9b-68a4-4095-ae36-4ca530d20c22 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:39.476069 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 230/464] 10.222.0.16 () {56 vars in 1137 bytes} [Sun Oct 30 00:43:39 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:39.477120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ed0c9b-68a4-4095-ae36-4ca530d20c22 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:39.509213 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377a0298-78ee-4604-a6c8-347a1aab2a69 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:39.527250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:39.547172 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1e068d-fc6e-41e1-93ee-cc6b36895e53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:39.547982 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1e068d-fc6e-41e1-93ee-cc6b36895e53 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:39.548351 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1e068d-fc6e-41e1-93ee-cc6b36895e53 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:39.548623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1e068d-fc6e-41e1-93ee-cc6b36895e53 None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:39.549299 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1e068d-fc6e-41e1-93ee-cc6b36895e53 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:39.549655 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1e068d-fc6e-41e1-93ee-cc6b36895e53 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:39.551061 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1e068d-fc6e-41e1-93ee-cc6b36895e53 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:39.567554 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 235/465] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:39.567554 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1e068d-fc6e-41e1-93ee-cc6b36895e53 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:39.593739 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ed0c9b-68a4-4095-ae36-4ca530d20c22 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:39.628711 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76333cbc-66e5-4c04-8548-f955fe7198b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:39.629381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76333cbc-66e5-4c04-8548-f955fe7198b0 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:39.629743 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76333cbc-66e5-4c04-8548-f955fe7198b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:39.630107 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76333cbc-66e5-4c04-8548-f955fe7198b0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:39.630919 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76333cbc-66e5-4c04-8548-f955fe7198b0 None admin] RBAC: Adding query filter params (role.id=5a0f6f642a10411da6fc1bfc92a7a3d3, scope.project.id=658799ed31d2472fb36af4aff0f03aab, user.id=e7a886715b5b4c1d99f0b37c58cb5417) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:39.631344 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76333cbc-66e5-4c04-8548-f955fe7198b0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:39.632917 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76333cbc-66e5-4c04-8548-f955fe7198b0 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:39.651942 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 231/466] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:43:39 2022] GET /identity/v3/role_assignments?user.id=e7a886715b5b4c1d99f0b37c58cb5417&scope.project.id=658799ed31d2472fb36af4aff0f03aab&role.id=5a0f6f642a10411da6fc1bfc92a7a3d3 => generated 608 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:39.653100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76333cbc-66e5-4c04-8548-f955fe7198b0 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:39.672158 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1e068d-fc6e-41e1-93ee-cc6b36895e53 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:39.697071 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:39.713721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857d8acd-60aa-46c9-8e05-f718eed134b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:39.714338 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857d8acd-60aa-46c9-8e05-f718eed134b7 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:39.714611 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857d8acd-60aa-46c9-8e05-f718eed134b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:39.714876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857d8acd-60aa-46c9-8e05-f718eed134b7 None admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:39.716162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857d8acd-60aa-46c9-8e05-f718eed134b7 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:39.716162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857d8acd-60aa-46c9-8e05-f718eed134b7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:39.717479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857d8acd-60aa-46c9-8e05-f718eed134b7 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:39.740926 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 236/467] 10.222.0.16 () {56 vars in 1137 bytes} [Sun Oct 30 00:43:39 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:39.741811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857d8acd-60aa-46c9-8e05-f718eed134b7 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:39.779887 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:39.795706 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76333cbc-66e5-4c04-8548-f955fe7198b0 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:39.840250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8eefd37-ec14-4905-bf72-34ea00112271 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:39.841025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eefd37-ec14-4905-bf72-34ea00112271 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:39.842162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eefd37-ec14-4905-bf72-34ea00112271 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:39.842993 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eefd37-ec14-4905-bf72-34ea00112271 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:39.862044 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857d8acd-60aa-46c9-8e05-f718eed134b7 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:39.901594 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a031f23-a8ce-48c7-b63f-1570b19a1763 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:39.902383 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a031f23-a8ce-48c7-b63f-1570b19a1763 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:39.902665 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a031f23-a8ce-48c7-b63f-1570b19a1763 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:39.902932 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a031f23-a8ce-48c7-b63f-1570b19a1763 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:39.913068 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a031f23-a8ce-48c7-b63f-1570b19a1763 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:39.915168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a031f23-a8ce-48c7-b63f-1570b19a1763 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:39.924178 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8eefd37-ec14-4905-bf72-34ea00112271 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:39.925854 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8eefd37-ec14-4905-bf72-34ea00112271 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:39.932670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-7a031f23-a8ce-48c7-b63f-1570b19a1763 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 30 00:43:39.932670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:39.932670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:39.932670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:39.932670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:39.932670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:39.932670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:39.932670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:39.932670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:39.932670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:39.932670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:39.932670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:39.932670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:43:39.932670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:43:39.932670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:43:39.932670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:43:39.933742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:39.933742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:39.933742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:43:39.933742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:39.933742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:43:39.933742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:39.933742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:39.933742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:39.933742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:39.933742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:39.933742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:39.933742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:39.933742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:39.933742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:39.933742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:39.933742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:39.933742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:39.933742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:39.934748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:39.934748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:39.934748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:43:39.934748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:43:39.934748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:43:39.934748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:43:39.934748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:43:39.934748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:43:39.934748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 30 00:43:39.934748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:39.934748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 237/468] 10.222.0.16 () {56 vars in 1102 bytes} [Sun Oct 30 00:43:39 2022] GET /identity/v3/users/neutron => generated 85 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:39.934748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a031f23-a8ce-48c7-b63f-1570b19a1763 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:39.955100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:40.040026 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:40.043216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 232/469] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:43:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 30 00:43:40.044233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8eefd37-ec14-4905-bf72-34ea00112271 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:40.048106 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a031f23-a8ce-48c7-b63f-1570b19a1763 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:40.083694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c82e236-0a08-473f-ae5c-63aa00065556 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:40.084506 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c82e236-0a08-473f-ae5c-63aa00065556 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:40.084881 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c82e236-0a08-473f-ae5c-63aa00065556 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:40.085245 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c82e236-0a08-473f-ae5c-63aa00065556 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:40.152517 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8eefd37-ec14-4905-bf72-34ea00112271 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:40.154602 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c82e236-0a08-473f-ae5c-63aa00065556 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:40.156219 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c82e236-0a08-473f-ae5c-63aa00065556 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:40.177535 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:40.189984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d278c52-602a-41e3-95e7-5fefdd89f2bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:40.190856 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d278c52-602a-41e3-95e7-5fefdd89f2bb None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:40.191364 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d278c52-602a-41e3-95e7-5fefdd89f2bb None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:40.191797 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d278c52-602a-41e3-95e7-5fefdd89f2bb None admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:40.192645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d278c52-602a-41e3-95e7-5fefdd89f2bb None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:40.193086 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d278c52-602a-41e3-95e7-5fefdd89f2bb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:40.194612 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d278c52-602a-41e3-95e7-5fefdd89f2bb None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:40.214405 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 233/470] 10.222.0.16 () {56 vars in 1139 bytes} [Sun Oct 30 00:43:40 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:40.215553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d278c52-602a-41e3-95e7-5fefdd89f2bb None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:40.259265 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:40.268337 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 238/471] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:43:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 30 00:43:40.269640 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c82e236-0a08-473f-ae5c-63aa00065556 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:40.336264 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d278c52-602a-41e3-95e7-5fefdd89f2bb None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:40.379040 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d3d29d-dab5-4d0b-8d81-d2e2a4f91b96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:40.379895 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d3d29d-dab5-4d0b-8d81-d2e2a4f91b96 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:40.380235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d3d29d-dab5-4d0b-8d81-d2e2a4f91b96 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:40.380597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d3d29d-dab5-4d0b-8d81-d2e2a4f91b96 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:40.392085 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d3d29d-dab5-4d0b-8d81-d2e2a4f91b96 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:40.393790 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d3d29d-dab5-4d0b-8d81-d2e2a4f91b96 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:40.406083 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c82e236-0a08-473f-ae5c-63aa00065556 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:40.411061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-50d3d29d-dab5-4d0b-8d81-d2e2a4f91b96 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:40.411061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:40.411061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:40.411061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:40.411061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:40.411061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:40.411061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:40.411061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:40.411061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:40.411061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:40.411061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:40.411061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:40.411061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:40.411061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:40.411061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:40.411061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:40.412141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:40.412141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:40.412141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:40.412141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:40.412141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:40.412141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:40.412141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:40.412141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:40.412141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:40.412141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:40.412141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:40.412141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:40.412141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:40.412141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:40.412141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:40.412141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:40.412141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:40.413215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:40.413215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:40.413215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:40.418890 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 234/472] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:40.419961 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d3d29d-dab5-4d0b-8d81-d2e2a4f91b96 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:40.451748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34cb4e2-0f96-4377-835e-458d8be7f140 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:40.452887 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34cb4e2-0f96-4377-835e-458d8be7f140 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:40.453161 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34cb4e2-0f96-4377-835e-458d8be7f140 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:40.453427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34cb4e2-0f96-4377-835e-458d8be7f140 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:40.463665 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:40.516435 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:40.524544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34cb4e2-0f96-4377-835e-458d8be7f140 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:40.525994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34cb4e2-0f96-4377-835e-458d8be7f140 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:40.547992 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d3d29d-dab5-4d0b-8d81-d2e2a4f91b96 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:40.588743 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41176136-40f3-4dff-a653-2d1cc101826c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:40.589971 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41176136-40f3-4dff-a653-2d1cc101826c None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:40.590418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41176136-40f3-4dff-a653-2d1cc101826c None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:40.590855 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41176136-40f3-4dff-a653-2d1cc101826c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:40.604501 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41176136-40f3-4dff-a653-2d1cc101826c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:40.606646 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41176136-40f3-4dff-a653-2d1cc101826c None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:40.624754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-41176136-40f3-4dff-a653-2d1cc101826c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:40.624754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:40.624754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:40.624754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:40.624754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:40.624754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:40.624754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:40.624754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:40.624754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:40.624754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:40.624754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:40.624754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:40.624754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:40.624754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:40.624754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:40.624754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:40.625832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:40.625832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:40.625832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:40.625832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:40.625832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:40.625832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:40.625832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:40.625832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:40.625832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:40.625832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:40.625832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:40.625832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:40.625832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:40.625832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:40.625832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:40.625832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:40.625832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:40.626657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:40.626657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:40.626657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:40.628940 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 235/473] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:40.630127 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41176136-40f3-4dff-a653-2d1cc101826c None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:40.634396 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 239/474] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:43:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 30 00:43:40.635401 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34cb4e2-0f96-4377-835e-458d8be7f140 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:40.730720 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:40.737731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34cb4e2-0f96-4377-835e-458d8be7f140 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:40.743636 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:40.762339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41176136-40f3-4dff-a653-2d1cc101826c None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:40.776881 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343156c5-873b-4707-b854-cde76a916aba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:40.778194 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343156c5-873b-4707-b854-cde76a916aba None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:40.778611 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343156c5-873b-4707-b854-cde76a916aba None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:40.779255 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343156c5-873b-4707-b854-cde76a916aba None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:40.780070 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343156c5-873b-4707-b854-cde76a916aba None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:40.780744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343156c5-873b-4707-b854-cde76a916aba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:40.782650 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343156c5-873b-4707-b854-cde76a916aba None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:40.798673 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 240/475] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:40.799923 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343156c5-873b-4707-b854-cde76a916aba None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:40.805400 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3775b363-1286-4efc-85ab-06173ae51fb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:40.806114 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3775b363-1286-4efc-85ab-06173ae51fb5 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:40.806525 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3775b363-1286-4efc-85ab-06173ae51fb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:40.806885 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3775b363-1286-4efc-85ab-06173ae51fb5 None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:40.807615 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3775b363-1286-4efc-85ab-06173ae51fb5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:40.808030 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3775b363-1286-4efc-85ab-06173ae51fb5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:40.809499 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3775b363-1286-4efc-85ab-06173ae51fb5 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:40.827761 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 236/476] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:40.828778 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3775b363-1286-4efc-85ab-06173ae51fb5 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:40.917855 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343156c5-873b-4707-b854-cde76a916aba None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:40.946543 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3775b363-1286-4efc-85ab-06173ae51fb5 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:40.953233 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b46cbd-2430-45d9-a6cc-50298eec31de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:40.954093 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b46cbd-2430-45d9-a6cc-50298eec31de None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:40.954507 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b46cbd-2430-45d9-a6cc-50298eec31de None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:40.955047 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b46cbd-2430-45d9-a6cc-50298eec31de None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:40.966573 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b46cbd-2430-45d9-a6cc-50298eec31de None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:40.968209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b46cbd-2430-45d9-a6cc-50298eec31de None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:40.982967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-38b46cbd-2430-45d9-a6cc-50298eec31de None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:40.982967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:40.982967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:40.982967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:40.982967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:40.982967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:40.982967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:40.982967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:40.982967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:40.982967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:40.982967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:40.982967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:40.982967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:40.982967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:40.982967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:40.982967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:40.983876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:40.983876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:40.983876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:40.983876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:40.983876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:40.983876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:40.983876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:40.983876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:40.983876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:40.983876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:40.983876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:40.983876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:40.983876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:40.983876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:40.983876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:40.983876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:40.983876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:40.984712 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:40.984712 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:40.984712 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:40.984712 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 241/477] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:40.985402 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b46cbd-2430-45d9-a6cc-50298eec31de None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:40.989946 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0ec6c3-3943-4a8b-8415-422fd4404be1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:40.990108 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0ec6c3-3943-4a8b-8415-422fd4404be1 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:40.990382 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0ec6c3-3943-4a8b-8415-422fd4404be1 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:40.990655 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0ec6c3-3943-4a8b-8415-422fd4404be1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:40.995585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:40.998421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0ec6c3-3943-4a8b-8415-422fd4404be1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:40.999906 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0ec6c3-3943-4a8b-8415-422fd4404be1 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:41.012789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-bd0ec6c3-3943-4a8b-8415-422fd4404be1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:41.012789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:41.012789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:41.012789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:41.012789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:41.012789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:41.012789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:41.012789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:41.012789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:41.012789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:41.012789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:41.012789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:41.012789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:43:41.012789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:43:41.012789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:43:41.013617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:43:41.013617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:41.013617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:41.013617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:41.013617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:41.013617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:41.013617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:41.013617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:41.013617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:41.013617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:41.013617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:41.013617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:41.013617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:41.013617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:41.013617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:41.013617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:41.013617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:41.014435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:43:41.014435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:43:41.014435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:43:41.014435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:43:41.014435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:43:41.014435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:43:41.014435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:41.014435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:41.016173 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 237/478] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:43:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:41.017447 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6c2aeb-8a1a-43dc-86eb-8f302cda0c78 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:41.017945 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6c2aeb-8a1a-43dc-86eb-8f302cda0c78 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:41.018378 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6c2aeb-8a1a-43dc-86eb-8f302cda0c78 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:41.019732 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 238/479] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:41.020532 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6c2aeb-8a1a-43dc-86eb-8f302cda0c78 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:41.030072 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:41.101807 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b46cbd-2430-45d9-a6cc-50298eec31de None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:41.128012 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6c2aeb-8a1a-43dc-86eb-8f302cda0c78 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:41.138750 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27f0e2c-bcc0-4027-b67b-46f7e883941c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:41.139382 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27f0e2c-bcc0-4027-b67b-46f7e883941c None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:41.139659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27f0e2c-bcc0-4027-b67b-46f7e883941c None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:41.139960 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27f0e2c-bcc0-4027-b67b-46f7e883941c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:41.147400 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e27f0e2c-bcc0-4027-b67b-46f7e883941c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:41.148973 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e27f0e2c-bcc0-4027-b67b-46f7e883941c None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:41.161250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-e27f0e2c-bcc0-4027-b67b-46f7e883941c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:41.161250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:41.161250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:41.161250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:41.161250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:41.161250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:41.161250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:41.161250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:41.161250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:41.161250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:41.161250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:41.161250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:41.161250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:43:41.161250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:43:41.161250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:43:41.162053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:43:41.162053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:41.162053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:41.162053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:41.162053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:41.162053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:41.162053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:41.162053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:41.162053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:41.162053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:41.162053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:41.162053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:41.162053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:41.162053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:41.162053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:41.162053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:41.162053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:41.162902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:43:41.162902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:43:41.162902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:43:41.162902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:43:41.162902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:43:41.162902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:43:41.162902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:41.162902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:41.163303 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0daa927e-bb2d-41a3-992b-fbd5f421a99d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:41.163303 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daa927e-bb2d-41a3-992b-fbd5f421a99d None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:41.163303 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daa927e-bb2d-41a3-992b-fbd5f421a99d None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:41.163518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 242/480] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:43:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:41.163584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daa927e-bb2d-41a3-992b-fbd5f421a99d None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:41.163584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0daa927e-bb2d-41a3-992b-fbd5f421a99d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:41.163706 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27f0e2c-bcc0-4027-b67b-46f7e883941c None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:41.163794 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0daa927e-bb2d-41a3-992b-fbd5f421a99d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:41.169291 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0daa927e-bb2d-41a3-992b-fbd5f421a99d None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:41.181361 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 239/481] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:41.187157 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff9f732-16b5-401a-bf9e-d76e44421d8a None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:41.187438 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff9f732-16b5-401a-bf9e-d76e44421d8a None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:41.187704 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff9f732-16b5-401a-bf9e-d76e44421d8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:41.206627 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-aff9f732-16b5-401a-bf9e-d76e44421d8a None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:41.207837 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-aff9f732-16b5-401a-bf9e-d76e44421d8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:41.240150 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:41.269517 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:41.276307 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27f0e2c-bcc0-4027-b67b-46f7e883941c None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:41.278106 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 240/482] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:41.278818 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff9f732-16b5-401a-bf9e-d76e44421d8a None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:41.330856 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0020ca46-7dee-4a35-a6f1-74a358002549 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:41.331455 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0020ca46-7dee-4a35-a6f1-74a358002549 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:41.331752 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0020ca46-7dee-4a35-a6f1-74a358002549 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:41.332013 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0020ca46-7dee-4a35-a6f1-74a358002549 None admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:41.332700 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0020ca46-7dee-4a35-a6f1-74a358002549 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:41.333004 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0020ca46-7dee-4a35-a6f1-74a358002549 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:41.334425 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0020ca46-7dee-4a35-a6f1-74a358002549 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:41.354972 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 243/483] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:43:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:41.355894 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0020ca46-7dee-4a35-a6f1-74a358002549 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:41.400405 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff9f732-16b5-401a-bf9e-d76e44421d8a None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:41.435409 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf4cff5-2ed1-4796-8cf9-a0c3fd63cae9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:41.436350 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf4cff5-2ed1-4796-8cf9-a0c3fd63cae9 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:41.436620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf4cff5-2ed1-4796-8cf9-a0c3fd63cae9 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:41.436879 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf4cff5-2ed1-4796-8cf9-a0c3fd63cae9 None admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:41.437563 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf4cff5-2ed1-4796-8cf9-a0c3fd63cae9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:41.438095 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf4cff5-2ed1-4796-8cf9-a0c3fd63cae9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:41.439593 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf4cff5-2ed1-4796-8cf9-a0c3fd63cae9 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:41.455929 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 241/484] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:43:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:41.455929 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf4cff5-2ed1-4796-8cf9-a0c3fd63cae9 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:41.496161 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0020ca46-7dee-4a35-a6f1-74a358002549 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:41.516196 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:41.531395 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:41.542281 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f9f804-57a4-473f-8892-f4990f5c5dc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:41.542915 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f9f804-57a4-473f-8892-f4990f5c5dc1 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:41.543192 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f9f804-57a4-473f-8892-f4990f5c5dc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:41.543460 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f9f804-57a4-473f-8892-f4990f5c5dc1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:41.550812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f9f804-57a4-473f-8892-f4990f5c5dc1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:41.552281 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f9f804-57a4-473f-8892-f4990f5c5dc1 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:41.563744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-53f9f804-57a4-473f-8892-f4990f5c5dc1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:41.563744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:41.563744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:41.563744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:41.563744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:41.563744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:41.563744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:41.563744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:41.563744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:41.563744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:41.563744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:41.563744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:41.563744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:43:41.563744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:43:41.563744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:43:41.564809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:43:41.564809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:41.564809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:41.564809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:41.564809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:41.564809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:41.564809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:41.564809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:41.564809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:41.564809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:41.564809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:41.564809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:41.564809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:41.564809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:41.564809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:41.564809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:41.564809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:41.565797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:43:41.565797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:43:41.565797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:43:41.565797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:43:41.565797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:43:41.565797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:43:41.565797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:41.565797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:41.565797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 244/485] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:43:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:41.565797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f9f804-57a4-473f-8892-f4990f5c5dc1 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:41.567271 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf4cff5-2ed1-4796-8cf9-a0c3fd63cae9 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:41.603504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bad33d4-d482-42ad-bc5e-b47195a1940f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:41.604357 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bad33d4-d482-42ad-bc5e-b47195a1940f None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:41.604627 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bad33d4-d482-42ad-bc5e-b47195a1940f None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:41.604889 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bad33d4-d482-42ad-bc5e-b47195a1940f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:41.605619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bad33d4-d482-42ad-bc5e-b47195a1940f None admin] RBAC: Adding query filter params (role.id=5a0f6f642a10411da6fc1bfc92a7a3d3, scope.project.id=658799ed31d2472fb36af4aff0f03aab, user.id=807b123652d245269336fcffaf36478e) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:41.605913 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bad33d4-d482-42ad-bc5e-b47195a1940f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:41.607411 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bad33d4-d482-42ad-bc5e-b47195a1940f None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:41.613560 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 242/486] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:43:41 2022] GET /identity/v3/role_assignments?user.id=807b123652d245269336fcffaf36478e&scope.project.id=658799ed31d2472fb36af4aff0f03aab&role.id=5a0f6f642a10411da6fc1bfc92a7a3d3 => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:41.614371 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bad33d4-d482-42ad-bc5e-b47195a1940f None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:41.690966 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f9f804-57a4-473f-8892-f4990f5c5dc1 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:41.729580 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6817350a-1a71-4397-a81c-41c4944f2be2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:41.730197 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6817350a-1a71-4397-a81c-41c4944f2be2 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:41.730470 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6817350a-1a71-4397-a81c-41c4944f2be2 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:41.730732 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6817350a-1a71-4397-a81c-41c4944f2be2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:41.731478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6817350a-1a71-4397-a81c-41c4944f2be2 None admin] RBAC: Adding query filter params (role.id=5a0f6f642a10411da6fc1bfc92a7a3d3, scope.project.id=658799ed31d2472fb36af4aff0f03aab, user.id=d57b1c923dff462ab9aaf238f95c6b7f) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:41.732926 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6817350a-1a71-4397-a81c-41c4944f2be2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:41.733409 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6817350a-1a71-4397-a81c-41c4944f2be2 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:41.745939 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bad33d4-d482-42ad-bc5e-b47195a1940f None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:41.748480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 245/487] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:43:41 2022] GET /identity/v3/role_assignments?user.id=d57b1c923dff462ab9aaf238f95c6b7f&scope.project.id=658799ed31d2472fb36af4aff0f03aab&role.id=5a0f6f642a10411da6fc1bfc92a7a3d3 => generated 608 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:41.750247 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6817350a-1a71-4397-a81c-41c4944f2be2 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:41.770317 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:41.786908 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1890d4-3e19-48be-a711-bdfe6d6e2c30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:41.787969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1890d4-3e19-48be-a711-bdfe6d6e2c30 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:41.788469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1890d4-3e19-48be-a711-bdfe6d6e2c30 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:41.788950 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1890d4-3e19-48be-a711-bdfe6d6e2c30 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:41.802741 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1890d4-3e19-48be-a711-bdfe6d6e2c30 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:41.808733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-8a1890d4-3e19-48be-a711-bdfe6d6e2c30 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:41.808733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:41.808733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:41.808733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:41.808733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:41.808733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:41.808733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:41.808733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:41.808733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:41.808733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:41.808733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:41.808733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:41.808733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:41.808733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:41.808733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:41.808733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:41.809714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:41.809714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:41.809714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:41.809714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:41.809714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:41.809714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:41.809714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:41.809714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:41.809714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:41.809714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:41.809714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:41.809714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:41.809714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:41.809714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:41.809714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:41.809714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:41.810487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:41.810487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:41.810487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:41.810487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:41.810487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:41.810487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:41.810487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:41.810487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:41.810487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:41.810487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:41.810487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:41.810487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:41.813853 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 243/488] 10.222.0.16 () {56 vars in 1102 bytes} [Sun Oct 30 00:43:41 2022] GET /identity/v3/roles/service => generated 85 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:41.821314 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1890d4-3e19-48be-a711-bdfe6d6e2c30 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:41.857511 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:41.920928 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6817350a-1a71-4397-a81c-41c4944f2be2 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:41.944821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1890d4-3e19-48be-a711-bdfe6d6e2c30 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:41.967678 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5b4b80-ef7f-4639-a378-73e0f369222f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:41.968321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5b4b80-ef7f-4639-a378-73e0f369222f None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:41.968594 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5b4b80-ef7f-4639-a378-73e0f369222f None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:41.968859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5b4b80-ef7f-4639-a378-73e0f369222f None admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:41.970085 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e5b4b80-ef7f-4639-a378-73e0f369222f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:41.970085 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e5b4b80-ef7f-4639-a378-73e0f369222f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:41.971436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e5b4b80-ef7f-4639-a378-73e0f369222f None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:41.985602 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 246/489] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:43:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:41.986950 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af880e63-3b7d-456c-82f0-dacf10062d70 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:41.988003 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c319c0c-9335-46a4-8fa3-f698362c1dfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:41.988570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c319c0c-9335-46a4-8fa3-f698362c1dfc None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:41.988840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c319c0c-9335-46a4-8fa3-f698362c1dfc None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:41.989105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c319c0c-9335-46a4-8fa3-f698362c1dfc None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:41.989757 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c319c0c-9335-46a4-8fa3-f698362c1dfc None admin] RBAC: Adding query filter params (name=service) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:41.990055 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c319c0c-9335-46a4-8fa3-f698362c1dfc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:41.991418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c319c0c-9335-46a4-8fa3-f698362c1dfc None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:41.992134 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af880e63-3b7d-456c-82f0-dacf10062d70 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:41.992413 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af880e63-3b7d-456c-82f0-dacf10062d70 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:41.993818 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 247/490] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:41 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:41.994564 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af880e63-3b7d-456c-82f0-dacf10062d70 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:42.004429 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 244/491] 10.222.0.16 () {56 vars in 1103 bytes} [Sun Oct 30 00:43:41 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:42.005911 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6ce2cc-2ffe-4387-908a-e1a67db265fb None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:42.006325 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6ce2cc-2ffe-4387-908a-e1a67db265fb None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:42.007920 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6ce2cc-2ffe-4387-908a-e1a67db265fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:42.028796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-0b6ce2cc-2ffe-4387-908a-e1a67db265fb None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:42.029979 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-0b6ce2cc-2ffe-4387-908a-e1a67db265fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:42.067453 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:42.106898 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 245/492] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:42.108533 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6ce2cc-2ffe-4387-908a-e1a67db265fb None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:42.111611 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af880e63-3b7d-456c-82f0-dacf10062d70 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:42.123514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:42.154539 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7086f827-0268-4f0b-8add-9321773efde6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:42.155139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7086f827-0268-4f0b-8add-9321773efde6 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:42.155473 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7086f827-0268-4f0b-8add-9321773efde6 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:42.155771 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7086f827-0268-4f0b-8add-9321773efde6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:42.156554 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7086f827-0268-4f0b-8add-9321773efde6 None admin] RBAC: Adding query filter params (role.id=5a0f6f642a10411da6fc1bfc92a7a3d3, scope.project.id=658799ed31d2472fb36af4aff0f03aab, user.id=81999329edfb4d4cac567d88d1f52909) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:42.157353 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7086f827-0268-4f0b-8add-9321773efde6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:42.159327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7086f827-0268-4f0b-8add-9321773efde6 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:42.168913 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 248/493] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:43:41 2022] GET /identity/v3/role_assignments?user.id=81999329edfb4d4cac567d88d1f52909&scope.project.id=658799ed31d2472fb36af4aff0f03aab&role.id=5a0f6f642a10411da6fc1bfc92a7a3d3 => generated 608 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:42.226671 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6ce2cc-2ffe-4387-908a-e1a67db265fb None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:42.265125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55cee254-c3bb-408a-81fc-cc2d38079901 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:42.265682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cee254-c3bb-408a-81fc-cc2d38079901 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:42.265959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cee254-c3bb-408a-81fc-cc2d38079901 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:42.266226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cee254-c3bb-408a-81fc-cc2d38079901 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:42.277927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55cee254-c3bb-408a-81fc-cc2d38079901 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:42.279502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55cee254-c3bb-408a-81fc-cc2d38079901 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:42.295218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-55cee254-c3bb-408a-81fc-cc2d38079901 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:42.295218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:42.295218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:42.295218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:42.295218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:42.295218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:42.295218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:42.295218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:42.295218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:42.295218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:42.295218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:42.295218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:42.295218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:42.295218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:42.295218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:42.295218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:42.296600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:42.296600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:42.296600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:42.296600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:42.296600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:42.296600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:42.296600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:42.296600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:42.296600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:42.296600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:42.296600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:42.296600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:42.296600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:42.296600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:42.296600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:42.296600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:42.296600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:42.298053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:42.298053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:42.298053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:42.298053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 246/494] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:42.304616 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7086f827-0268-4f0b-8add-9321773efde6 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:42.448946 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7086f827-0268-4f0b-8add-9321773efde6 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:42.455076 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55cee254-c3bb-408a-81fc-cc2d38079901 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:42.484622 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:42.485447 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d136c562-8641-4762-9dcb-a040f681a065 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:42.486058 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d136c562-8641-4762-9dcb-a040f681a065 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:42.487543 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d136c562-8641-4762-9dcb-a040f681a065 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:42.487803 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d136c562-8641-4762-9dcb-a040f681a065 None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:42.488467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d136c562-8641-4762-9dcb-a040f681a065 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:42.488762 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d136c562-8641-4762-9dcb-a040f681a065 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:42.490272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d136c562-8641-4762-9dcb-a040f681a065 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:42.506723 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 249/495] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:42.512668 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d136c562-8641-4762-9dcb-a040f681a065 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:42.573502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:42.624481 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55cee254-c3bb-408a-81fc-cc2d38079901 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:42.660868 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10104289-970c-4d03-a735-116f54e14ae7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:42.661512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10104289-970c-4d03-a735-116f54e14ae7 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:42.661786 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10104289-970c-4d03-a735-116f54e14ae7 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:42.662052 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10104289-970c-4d03-a735-116f54e14ae7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:42.685072 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d136c562-8641-4762-9dcb-a040f681a065 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:42.692713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10104289-970c-4d03-a735-116f54e14ae7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:42.698024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-10104289-970c-4d03-a735-116f54e14ae7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:43:42.698024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:42.698024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:42.698024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:42.698024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:42.698024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:42.698024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:42.698024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:42.698024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:42.698024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:42.698024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:42.698024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:42.698024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:42.698024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:42.698024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:42.698024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:42.698912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:42.698912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:42.698912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:42.698912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:42.698912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:42.698912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:42.698912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:42.698912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:42.698912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:42.698912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:42.698912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:42.698912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:42.698912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:42.698912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:42.698912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:42.698912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:42.699839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:42.699839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:42.699839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:42.699839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:42.699839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:42.699839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:42.699839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:42.699839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:42.699839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:42.699839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:42.699839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:43:42.699839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:42.699839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 247/496] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:43:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:43:42.706164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10104289-970c-4d03-a735-116f54e14ae7 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:42.731038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c79cab8-429b-4c23-b1c3-f776a0636d05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:42.731741 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c79cab8-429b-4c23-b1c3-f776a0636d05 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:42.731988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c79cab8-429b-4c23-b1c3-f776a0636d05 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:42.732564 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c79cab8-429b-4c23-b1c3-f776a0636d05 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:42.741581 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c79cab8-429b-4c23-b1c3-f776a0636d05 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:42.743336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c79cab8-429b-4c23-b1c3-f776a0636d05 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:42.758480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-0c79cab8-429b-4c23-b1c3-f776a0636d05 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 30 00:43:42.758480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:42.758480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:42.758480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:42.758480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:42.758480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:42.758480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:42.758480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:42.758480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:42.758480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:42.758480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:42.758480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:42.758480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:43:42.758480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:43:42.758480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:43:42.758480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:43:42.759436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:42.759436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:42.759436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:43:42.759436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:42.759436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:43:42.759436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:42.759436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:42.759436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:42.759436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:42.759436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:42.759436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:42.759436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:42.759436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:42.759436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:42.759436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:42.759436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:42.759436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:42.759436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:42.760388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:42.760388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:42.760388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:43:42.760388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:43:42.760388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:43:42.760388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:43:42.760388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:43:42.760388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:43:42.760388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 30 00:43:42.760388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:42.760388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 250/497] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:43:42 2022] GET /identity/v3/users/swift => generated 83 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:42.774944 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c79cab8-429b-4c23-b1c3-f776a0636d05 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:42.832296 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:42.865125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10104289-970c-4d03-a735-116f54e14ae7 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:42.926802 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:42.932163 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01a8fff-882c-4173-863b-dcba150e8fe2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:42.932904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01a8fff-882c-4173-863b-dcba150e8fe2 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:42.933242 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01a8fff-882c-4173-863b-dcba150e8fe2 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:42.933428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01a8fff-882c-4173-863b-dcba150e8fe2 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:42.934097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01a8fff-882c-4173-863b-dcba150e8fe2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:42.934400 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01a8fff-882c-4173-863b-dcba150e8fe2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:42.936207 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01a8fff-882c-4173-863b-dcba150e8fe2 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:42.937105 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c79cab8-429b-4c23-b1c3-f776a0636d05 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:42.955845 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 248/498] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:43:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:42.961387 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01a8fff-882c-4173-863b-dcba150e8fe2 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:42.975305 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd6614a-600f-4751-a403-2d788804a957 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:42.976245 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd6614a-600f-4751-a403-2d788804a957 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:42.976741 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd6614a-600f-4751-a403-2d788804a957 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:42.977262 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd6614a-600f-4751-a403-2d788804a957 None admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:42.978169 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd6614a-600f-4751-a403-2d788804a957 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:42.978669 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd6614a-600f-4751-a403-2d788804a957 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:42.980650 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd6614a-600f-4751-a403-2d788804a957 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:42.999289 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 251/499] 10.222.0.16 () {56 vars in 1135 bytes} [Sun Oct 30 00:43:42 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:43.005172 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd6614a-600f-4751-a403-2d788804a957 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:43.137374 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01a8fff-882c-4173-863b-dcba150e8fe2 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:43.161155 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd6614a-600f-4751-a403-2d788804a957 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:43.204421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a348b314-f79c-45ab-9c0c-6bb59b0e7b28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:43.205057 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a348b314-f79c-45ab-9c0c-6bb59b0e7b28 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:43.205660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a348b314-f79c-45ab-9c0c-6bb59b0e7b28 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:43.206024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a348b314-f79c-45ab-9c0c-6bb59b0e7b28 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:43.223333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecbea7d-6465-4ca2-b634-de62a401f5fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:43.224021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecbea7d-6465-4ca2-b634-de62a401f5fd None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:43.224300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecbea7d-6465-4ca2-b634-de62a401f5fd None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:43.224571 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ecbea7d-6465-4ca2-b634-de62a401f5fd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:43.229887 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a348b314-f79c-45ab-9c0c-6bb59b0e7b28 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:43.231453 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a348b314-f79c-45ab-9c0c-6bb59b0e7b28 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:43.245851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:43.256861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-a348b314-f79c-45ab-9c0c-6bb59b0e7b28 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:43.256861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:43.256861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:43.256861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:43.256861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:43.256861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:43.256861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:43.256861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:43.256861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:43.256861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:43.256861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:43.256861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:43.256861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:43.256861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:43.256861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:43.256861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:43.258338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:43.258338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:43.258338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:43.258338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:43.258338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:43.258338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:43.258338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:43.258338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:43.258338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:43.258338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:43.258338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:43.258338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:43.258338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:43.258338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:43.258338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:43.258338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:43.258338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:43.259214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:43.259214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:43.259214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:43.259214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 249/500] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:43:43.263451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a348b314-f79c-45ab-9c0c-6bb59b0e7b28 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:43.269511 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:43.332536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecbea7d-6465-4ca2-b634-de62a401f5fd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:43.334326 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ecbea7d-6465-4ca2-b634-de62a401f5fd None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:43.483329 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 252/501] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:43:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 479 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 30 00:43:43.485434 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a348b314-f79c-45ab-9c0c-6bb59b0e7b28 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:43.489383 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecbea7d-6465-4ca2-b634-de62a401f5fd None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:43.549484 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d663fdf-9aa7-41fc-af71-93bcf3228d9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:43.553995 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d663fdf-9aa7-41fc-af71-93bcf3228d9f None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:43.554475 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d663fdf-9aa7-41fc-af71-93bcf3228d9f None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:43.554933 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d663fdf-9aa7-41fc-af71-93bcf3228d9f None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:43.555818 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d663fdf-9aa7-41fc-af71-93bcf3228d9f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:43.556309 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d663fdf-9aa7-41fc-af71-93bcf3228d9f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:43.557857 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d663fdf-9aa7-41fc-af71-93bcf3228d9f None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:43.580340 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:43.581296 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 250/502] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:43.586837 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d663fdf-9aa7-41fc-af71-93bcf3228d9f None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:43.665455 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ecbea7d-6465-4ca2-b634-de62a401f5fd None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:43.698100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:43.740521 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967fac93-dcf2-486c-be16-645dc2ff1721 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:43.741161 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967fac93-dcf2-486c-be16-645dc2ff1721 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:43.741353 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967fac93-dcf2-486c-be16-645dc2ff1721 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:43.741620 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967fac93-dcf2-486c-be16-645dc2ff1721 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:43.745790 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d663fdf-9aa7-41fc-af71-93bcf3228d9f None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:43.764807 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967fac93-dcf2-486c-be16-645dc2ff1721 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:43.766277 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967fac93-dcf2-486c-be16-645dc2ff1721 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:43.793209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-967fac93-dcf2-486c-be16-645dc2ff1721 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:43.793209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:43.793209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:43.793209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:43.793209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:43.793209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:43.793209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:43.793209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:43.793209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:43.793209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:43.793209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:43.793209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:43.793209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:43.793209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:43.793209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:43.793209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:43.794258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:43.794258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:43.794258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:43.794258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:43.794258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:43.794258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:43.794258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:43.794258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:43.794258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:43.794258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:43.794258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:43.794258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:43.794258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:43.794258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:43.794258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:43.794258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:43.794258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:43.795112 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:43.795112 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:43.795112 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:43.797024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea220947-f1f6-4236-b7e5-12a16e521666 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:43.797662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea220947-f1f6-4236-b7e5-12a16e521666 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:43.797934 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea220947-f1f6-4236-b7e5-12a16e521666 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:43.798197 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea220947-f1f6-4236-b7e5-12a16e521666 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:43.799094 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 253/503] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:43:43.800454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967fac93-dcf2-486c-be16-645dc2ff1721 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:43.817215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea220947-f1f6-4236-b7e5-12a16e521666 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:43.819379 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea220947-f1f6-4236-b7e5-12a16e521666 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:43.841770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-ea220947-f1f6-4236-b7e5-12a16e521666 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 30 00:43:43.841770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:43.841770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:43.841770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:43.841770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:43.841770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:43.841770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:43.841770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:43.841770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:43.841770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:43.841770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:43.841770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:43.841770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:43:43.841770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:43:43.841770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:43:43.841770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:43:43.842763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:43.842763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:43.842763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:43:43.842763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:43.842763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:43:43.842763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:43.842763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:43.842763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:43.842763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:43.842763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:43.842763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:43.842763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:43.842763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:43.842763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:43.842763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:43.842763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:43.842763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:43.842763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:43.843810 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:43.843810 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:43.843810 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:43:43.843810 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:43:43.843810 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:43:43.843810 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:43:43.843810 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:43:43.843810 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:43:43.843810 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 30 00:43:43.843810 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:43.847531 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 251/504] 10.222.0.16 () {56 vars in 1093 bytes} [Sun Oct 30 00:43:43 2022] GET /identity/v3/users/nova => generated 82 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:43.855445 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea220947-f1f6-4236-b7e5-12a16e521666 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:43.967237 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967fac93-dcf2-486c-be16-645dc2ff1721 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:43.969677 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:44.004734 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bece191-d4e4-4024-a182-3e80de2ee715 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:44.005342 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bece191-d4e4-4024-a182-3e80de2ee715 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:44.005616 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bece191-d4e4-4024-a182-3e80de2ee715 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:44.005887 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bece191-d4e4-4024-a182-3e80de2ee715 None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:44.006524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bece191-d4e4-4024-a182-3e80de2ee715 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:44.006818 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bece191-d4e4-4024-a182-3e80de2ee715 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:44.008250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bece191-d4e4-4024-a182-3e80de2ee715 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:44.022570 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 254/505] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:44.037710 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bece191-d4e4-4024-a182-3e80de2ee715 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:44.065961 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea220947-f1f6-4236-b7e5-12a16e521666 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:44.096081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:44.128436 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0f75cf-1872-48da-8daf-3477079628e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:44.129042 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0f75cf-1872-48da-8daf-3477079628e8 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:44.129316 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0f75cf-1872-48da-8daf-3477079628e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:44.129623 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0f75cf-1872-48da-8daf-3477079628e8 None admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:44.130445 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0f75cf-1872-48da-8daf-3477079628e8 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:44.130788 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0f75cf-1872-48da-8daf-3477079628e8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:44.132233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0f75cf-1872-48da-8daf-3477079628e8 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:44.157549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 252/506] 10.222.0.16 () {56 vars in 1133 bytes} [Sun Oct 30 00:43:43 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:44.164770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0f75cf-1872-48da-8daf-3477079628e8 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:44.202638 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bece191-d4e4-4024-a182-3e80de2ee715 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:44.237382 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb3ffde-c328-4c62-9e07-8a4b81fe730c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:44.238024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb3ffde-c328-4c62-9e07-8a4b81fe730c None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:44.238297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb3ffde-c328-4c62-9e07-8a4b81fe730c None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:44.238568 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb3ffde-c328-4c62-9e07-8a4b81fe730c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:44.246378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb3ffde-c328-4c62-9e07-8a4b81fe730c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:44.247884 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb3ffde-c328-4c62-9e07-8a4b81fe730c None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:44.260567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-fdb3ffde-c328-4c62-9e07-8a4b81fe730c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:44.260567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:44.260567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:44.260567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:44.260567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:44.260567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:44.260567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:44.260567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:44.260567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:44.260567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:44.260567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:44.260567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:44.260567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:43:44.260567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:43:44.260567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:43:44.261478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:43:44.261478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:44.261478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:44.261478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:44.261478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:44.261478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:44.261478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:44.261478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:44.261478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:44.261478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:44.261478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:44.261478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:44.261478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:44.261478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:44.261478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:44.261478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:44.261478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:44.262351 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:43:44.262351 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:43:44.262351 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:43:44.262351 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:43:44.262351 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:43:44.262351 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:43:44.262351 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:44.262351 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:44.262351 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 255/507] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:43:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:44.263302 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f871766f-07c8-4604-8cac-05111dfd2886 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:44.266954 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f871766f-07c8-4604-8cac-05111dfd2886 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:44.266954 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f871766f-07c8-4604-8cac-05111dfd2886 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:44.269475 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 256/508] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:44.270444 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f871766f-07c8-4604-8cac-05111dfd2886 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:44.271418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0f75cf-1872-48da-8daf-3477079628e8 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:44.297373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:44.312240 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0410cd3d-14ec-4acb-82ca-dce34d722bef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:44.312864 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0410cd3d-14ec-4acb-82ca-dce34d722bef None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:44.313135 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0410cd3d-14ec-4acb-82ca-dce34d722bef None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:44.313398 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0410cd3d-14ec-4acb-82ca-dce34d722bef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:44.324738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0410cd3d-14ec-4acb-82ca-dce34d722bef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:44.326277 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0410cd3d-14ec-4acb-82ca-dce34d722bef None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:44.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-0410cd3d-14ec-4acb-82ca-dce34d722bef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:44.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:44.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:44.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:44.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:44.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:44.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:44.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:44.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:44.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:44.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:44.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:44.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:44.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:44.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:44.344421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:44.346669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:44.346669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:44.346669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:44.346669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:44.346669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:44.346669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:44.346669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:44.346669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:44.346669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:44.346669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:44.346669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:44.346669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:44.346669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:44.346669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:44.346669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:44.346669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:44.346669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:44.348756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:44.348756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:44.348756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:44.355754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 253/509] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:44.355754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367a45f5-ed6b-4a52-bf54-0ac626005377 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:44.355754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367a45f5-ed6b-4a52-bf54-0ac626005377 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:44.355754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367a45f5-ed6b-4a52-bf54-0ac626005377 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:44.372526 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-367a45f5-ed6b-4a52-bf54-0ac626005377 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:44.373848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-367a45f5-ed6b-4a52-bf54-0ac626005377 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:44.377206 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f871766f-07c8-4604-8cac-05111dfd2886 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:44.399983 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:44.412070 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c059896-7e4b-4ab4-b619-c54877b4167e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:44.412686 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c059896-7e4b-4ab4-b619-c54877b4167e None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:44.412957 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c059896-7e4b-4ab4-b619-c54877b4167e None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:44.413221 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c059896-7e4b-4ab4-b619-c54877b4167e None admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:44.413908 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c059896-7e4b-4ab4-b619-c54877b4167e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:44.414207 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c059896-7e4b-4ab4-b619-c54877b4167e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:44.415616 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c059896-7e4b-4ab4-b619-c54877b4167e None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:44.427902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 257/510] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:43:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:44.427902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c059896-7e4b-4ab4-b619-c54877b4167e None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:44.456357 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 254/511] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:44.463902 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ceade2-85bd-4d30-b7e0-47617760323b None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:44.463902 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ceade2-85bd-4d30-b7e0-47617760323b None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:44.463902 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ceade2-85bd-4d30-b7e0-47617760323b None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:44.463902 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 255/512] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:44 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:44.463902 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ceade2-85bd-4d30-b7e0-47617760323b None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:44.542206 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:44.562903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c059896-7e4b-4ab4-b619-c54877b4167e None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:44.584817 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ceade2-85bd-4d30-b7e0-47617760323b None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:44.608170 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2835c93-cd00-4228-af32-c9ab2e95d15e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:44.608783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2835c93-cd00-4228-af32-c9ab2e95d15e None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:44.609055 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2835c93-cd00-4228-af32-c9ab2e95d15e None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:44.609322 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2835c93-cd00-4228-af32-c9ab2e95d15e None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:44.610485 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2835c93-cd00-4228-af32-c9ab2e95d15e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:44.610788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2835c93-cd00-4228-af32-c9ab2e95d15e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:44.612475 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2835c93-cd00-4228-af32-c9ab2e95d15e None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:44.626502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8cd922-9d06-48da-98f0-cc6b763258c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:44.631598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 258/513] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:44.631598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589d5e99-d6d7-41c8-8e9d-55b1daf35ddd None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:44.631598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589d5e99-d6d7-41c8-8e9d-55b1daf35ddd None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:44.631598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589d5e99-d6d7-41c8-8e9d-55b1daf35ddd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:44.632355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8cd922-9d06-48da-98f0-cc6b763258c6 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:44.632900 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8cd922-9d06-48da-98f0-cc6b763258c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:44.633337 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8cd922-9d06-48da-98f0-cc6b763258c6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:44.634235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8cd922-9d06-48da-98f0-cc6b763258c6 None admin] RBAC: Adding query filter params (role.id=5a0f6f642a10411da6fc1bfc92a7a3d3, scope.project.id=658799ed31d2472fb36af4aff0f03aab, user.id=67f6eff8c0254cc6a210101dd7af988d) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:44.634703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8cd922-9d06-48da-98f0-cc6b763258c6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:44.636425 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8cd922-9d06-48da-98f0-cc6b763258c6 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:44.643172 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 256/514] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:43:44 2022] GET /identity/v3/role_assignments?user.id=67f6eff8c0254cc6a210101dd7af988d&scope.project.id=658799ed31d2472fb36af4aff0f03aab&role.id=5a0f6f642a10411da6fc1bfc92a7a3d3 => generated 608 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:44.644353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8cd922-9d06-48da-98f0-cc6b763258c6 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:44.663338 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-589d5e99-d6d7-41c8-8e9d-55b1daf35ddd None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:44.664900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-589d5e99-d6d7-41c8-8e9d-55b1daf35ddd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:44.699048 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:44.779980 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 259/515] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:44.780890 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589d5e99-d6d7-41c8-8e9d-55b1daf35ddd None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:44.853128 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8cd922-9d06-48da-98f0-cc6b763258c6 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:44.897341 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08258a88-1279-4311-966b-44b3cc47a253 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:44.897967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08258a88-1279-4311-966b-44b3cc47a253 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:44.898238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08258a88-1279-4311-966b-44b3cc47a253 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:44.898500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08258a88-1279-4311-966b-44b3cc47a253 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:44.904229 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:44.908488 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08258a88-1279-4311-966b-44b3cc47a253 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:44.909811 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08258a88-1279-4311-966b-44b3cc47a253 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:44.923023 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-08258a88-1279-4311-966b-44b3cc47a253 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:44.923023 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:44.923023 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:44.923023 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:44.923023 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:44.923023 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:44.923023 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:44.923023 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:44.923023 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:44.923023 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:44.923023 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:44.923023 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:44.923023 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:43:44.923023 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:43:44.923023 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:43:44.924394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:43:44.924394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:44.924394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:44.924394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:44.924394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:44.924394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:44.924394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:44.924394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:44.924394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:44.924394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:44.924394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:44.924394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:44.924394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:44.924394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:44.924394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:44.924394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:44.924394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:44.925431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:43:44.925431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:43:44.925431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:43:44.925431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:43:44.925431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:43:44.925431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:43:44.925431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:44.925431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:44.925431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 257/516] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:43:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:44.925431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040f8e93-880e-4b96-94e0-90f311a43c43 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:44.925959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040f8e93-880e-4b96-94e0-90f311a43c43 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:44.925959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040f8e93-880e-4b96-94e0-90f311a43c43 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:44.927281 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 258/517] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:44 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:44.929994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040f8e93-880e-4b96-94e0-90f311a43c43 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:44.942167 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589d5e99-d6d7-41c8-8e9d-55b1daf35ddd None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:44.987225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de9c8df-58f3-4d12-8208-4918375d319b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:44.988444 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de9c8df-58f3-4d12-8208-4918375d319b None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:44.989712 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de9c8df-58f3-4d12-8208-4918375d319b None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:44.989983 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de9c8df-58f3-4d12-8208-4918375d319b None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:44.990800 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de9c8df-58f3-4d12-8208-4918375d319b None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:44.992256 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de9c8df-58f3-4d12-8208-4918375d319b None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:45.004553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-3de9c8df-58f3-4d12-8208-4918375d319b None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 30 00:43:45.004553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:45.004553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:43:45.004553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:43:45.004553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:43:45.004553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:43:45.004553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:43:45.004553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:43:45.004553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 30 00:43:45.004553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:45.004553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:43:45.004553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:45.004553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:45.004553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:45.004553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:45.004553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:45.004553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:45.005548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:45.005548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:45.005548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:45.005548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:45.005548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:45.005548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:45.005548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:43:45.005548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:43:45.005548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:43:45.005548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:43:45.005548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:45.005548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:45.005548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:45.005548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:45.005548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:45.005548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:45.006372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:45.006372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:45.006372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:45.006372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:45.006372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:45.006372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:45.006372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:45.006372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:45.006372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:45.006372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:45.006372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:43:45.006372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:43:45.006372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 30 00:43:45.006372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:45.006372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 260/518] 10.222.0.16 () {56 vars in 1129 bytes} [Sun Oct 30 00:43:44 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:45.011529 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a91f67c-6f8c-4afc-8dca-ffae6f32bbcb None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:45.011998 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a91f67c-6f8c-4afc-8dca-ffae6f32bbcb None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:45.012276 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a91f67c-6f8c-4afc-8dca-ffae6f32bbcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:45.024877 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:45.033757 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-1a91f67c-6f8c-4afc-8dca-ffae6f32bbcb None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:45.034903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-1a91f67c-6f8c-4afc-8dca-ffae6f32bbcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:45.048422 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040f8e93-880e-4b96-94e0-90f311a43c43 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:45.084765 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6313868-4449-4d68-b909-b725c26a913b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:45.085363 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6313868-4449-4d68-b909-b725c26a913b None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:45.085633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6313868-4449-4d68-b909-b725c26a913b None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:45.085895 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6313868-4449-4d68-b909-b725c26a913b None admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:45.086570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6313868-4449-4d68-b909-b725c26a913b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:45.086954 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6313868-4449-4d68-b909-b725c26a913b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:45.088416 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6313868-4449-4d68-b909-b725c26a913b None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:45.103857 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 259/519] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:43:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:45.103857 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6313868-4449-4d68-b909-b725c26a913b None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:45.106481 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 261/520] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:45.107466 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a91f67c-6f8c-4afc-8dca-ffae6f32bbcb None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:45.201180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:45.218219 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6313868-4449-4d68-b909-b725c26a913b None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:45.253584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a91f67c-6f8c-4afc-8dca-ffae6f32bbcb None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:45.259425 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a29695-acd8-4a08-9f33-3312d028ae44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:45.260133 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a29695-acd8-4a08-9f33-3312d028ae44 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:45.260467 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a29695-acd8-4a08-9f33-3312d028ae44 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:45.260793 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a29695-acd8-4a08-9f33-3312d028ae44 None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:45.261470 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a29695-acd8-4a08-9f33-3312d028ae44 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:45.261831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a29695-acd8-4a08-9f33-3312d028ae44 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:45.263252 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a29695-acd8-4a08-9f33-3312d028ae44 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:45.271755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 260/521] 10.222.0.16 () {56 vars in 1124 bytes} [Sun Oct 30 00:43:45 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:45.271755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a29695-acd8-4a08-9f33-3312d028ae44 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:45.283986 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:45.308172 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078aa72c-b941-4302-ac59-36cf0f96a5a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:45.309747 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078aa72c-b941-4302-ac59-36cf0f96a5a1 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:45.310222 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078aa72c-b941-4302-ac59-36cf0f96a5a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:45.311016 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078aa72c-b941-4302-ac59-36cf0f96a5a1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:45.324885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078aa72c-b941-4302-ac59-36cf0f96a5a1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:45.327366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078aa72c-b941-4302-ac59-36cf0f96a5a1 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:45.346456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-078aa72c-b941-4302-ac59-36cf0f96a5a1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:45.346456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:45.346456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:45.346456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:45.346456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:45.346456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:45.346456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:45.346456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:45.346456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:45.346456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:45.346456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:45.346456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:45.346456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:45.346456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:45.346456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:45.346456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:45.347434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:45.347434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:45.347434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:45.347434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:45.347434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:45.347434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:45.347434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:45.347434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:45.347434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:45.347434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:45.347434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:45.347434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:45.347434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:45.347434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:45.347434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:45.347434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:45.347434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:45.348360 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:45.348360 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:45.348360 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:45.348360 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 262/522] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:45.349006 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078aa72c-b941-4302-ac59-36cf0f96a5a1 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:45.397281 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a29695-acd8-4a08-9f33-3312d028ae44 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:45.431875 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af3bd58-5231-4a58-8eb9-97eb785ee6e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:45.432526 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af3bd58-5231-4a58-8eb9-97eb785ee6e3 None admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:45.432855 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af3bd58-5231-4a58-8eb9-97eb785ee6e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:45.433126 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af3bd58-5231-4a58-8eb9-97eb785ee6e3 None admin] PATH_INFO: `/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/e7a886715b5b4c1d99f0b37c58cb5417/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:45.435036 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af3bd58-5231-4a58-8eb9-97eb785ee6e3 None admin] RBAC: Authorizing `identity:create_grant(project_id=658799ed31d2472fb36af4aff0f03aab, user_id=e7a886715b5b4c1d99f0b37c58cb5417, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:45.436800 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af3bd58-5231-4a58-8eb9-97eb785ee6e3 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:45.452188 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 261/523] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:43:45 2022] PUT /identity/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/e7a886715b5b4c1d99f0b37c58cb5417/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:43:45.453139 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af3bd58-5231-4a58-8eb9-97eb785ee6e3 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:45.461493 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078aa72c-b941-4302-ac59-36cf0f96a5a1 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:45.493647 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:45.519037 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4cfe02-99e2-4a85-a6e7-34169fab66a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:45.519993 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4cfe02-99e2-4a85-a6e7-34169fab66a5 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:45.520468 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4cfe02-99e2-4a85-a6e7-34169fab66a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:45.521575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4cfe02-99e2-4a85-a6e7-34169fab66a5 None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:45.522676 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4cfe02-99e2-4a85-a6e7-34169fab66a5 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:45.523247 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4cfe02-99e2-4a85-a6e7-34169fab66a5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:45.525326 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4cfe02-99e2-4a85-a6e7-34169fab66a5 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:45.535310 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 263/524] 10.222.0.16 () {56 vars in 1124 bytes} [Sun Oct 30 00:43:45 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:45.536553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4cfe02-99e2-4a85-a6e7-34169fab66a5 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:45.546350 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:45.621786 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af3bd58-5231-4a58-8eb9-97eb785ee6e3 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:45.674015 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb764d7d-bd44-4287-9eb1-4378fe4c4235 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:45.674811 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb764d7d-bd44-4287-9eb1-4378fe4c4235 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:45.675191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb764d7d-bd44-4287-9eb1-4378fe4c4235 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:45.675462 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb764d7d-bd44-4287-9eb1-4378fe4c4235 None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:45.676150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb764d7d-bd44-4287-9eb1-4378fe4c4235 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:45.676601 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb764d7d-bd44-4287-9eb1-4378fe4c4235 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:45.678114 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb764d7d-bd44-4287-9eb1-4378fe4c4235 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:45.680813 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4cfe02-99e2-4a85-a6e7-34169fab66a5 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:45.695905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 262/525] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:45.698677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb764d7d-bd44-4287-9eb1-4378fe4c4235 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:45.721653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f62e838-f203-4db6-a1b9-36fc2db80315 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:45.722651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f62e838-f203-4db6-a1b9-36fc2db80315 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:45.723094 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f62e838-f203-4db6-a1b9-36fc2db80315 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:45.723529 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f62e838-f203-4db6-a1b9-36fc2db80315 None admin] PATH_INFO: `/v3/services/network` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:45.724384 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f62e838-f203-4db6-a1b9-36fc2db80315 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:45.726013 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f62e838-f203-4db6-a1b9-36fc2db80315 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:45.735684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-8f62e838-f203-4db6-a1b9-36fc2db80315 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 30 00:43:45.735684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:45.735684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:43:45.735684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:43:45.735684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:43:45.735684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:43:45.735684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:43:45.735684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:43:45.735684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 30 00:43:45.735684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:45.735684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:43:45.735684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:45.735684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:45.735684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:45.735684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:45.735684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:45.735684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:45.736872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:45.736872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:45.736872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:45.736872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:45.736872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:45.736872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:45.736872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:43:45.736872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:43:45.736872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:43:45.736872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:43:45.736872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:45.736872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:45.736872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:45.736872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:45.736872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:45.736872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:45.737702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:45.737702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:45.737702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:45.737702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:45.737702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:45.737702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:45.737702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:45.737702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:45.737702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:45.737702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:45.737702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:43:45.737702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:43:45.737702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 30 00:43:45.737702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:45.740590 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 264/526] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:43:45 2022] GET /identity/v3/services/network => generated 88 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:45.760988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f62e838-f203-4db6-a1b9-36fc2db80315 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:45.814306 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb764d7d-bd44-4287-9eb1-4378fe4c4235 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:45.836135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:45.841443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:45.851411 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180fcaa6-2e7d-4bf6-86f0-758757854590 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:45.851985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180fcaa6-2e7d-4bf6-86f0-758757854590 None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:45.852261 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180fcaa6-2e7d-4bf6-86f0-758757854590 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:45.852525 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180fcaa6-2e7d-4bf6-86f0-758757854590 None admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:45.853164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180fcaa6-2e7d-4bf6-86f0-758757854590 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:45.854805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180fcaa6-2e7d-4bf6-86f0-758757854590 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:45.880627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f62e838-f203-4db6-a1b9-36fc2db80315 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:45.887933 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 263/527] 10.222.0.16 () {60 vars in 1132 bytes} [Sun Oct 30 00:43:45 2022] POST /identity/v3/users => generated 254 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:43:45.922446 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac28887-a09b-43e2-8395-f05c6671fac2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:45.923475 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac28887-a09b-43e2-8395-f05c6671fac2 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:45.924182 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac28887-a09b-43e2-8395-f05c6671fac2 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:45.924331 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac28887-a09b-43e2-8395-f05c6671fac2 None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:45.925106 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac28887-a09b-43e2-8395-f05c6671fac2 None admin] RBAC: Adding query filter params (name=network) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:45.925407 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac28887-a09b-43e2-8395-f05c6671fac2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:45.926914 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac28887-a09b-43e2-8395-f05c6671fac2 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:45.932439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 265/528] 10.222.0.16 () {56 vars in 1112 bytes} [Sun Oct 30 00:43:45 2022] GET /identity/v3/services?name=network => generated 124 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:45.938216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180fcaa6-2e7d-4bf6-86f0-758757854590 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:46.080461 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180fcaa6-2e7d-4bf6-86f0-758757854590 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:46.122435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313ef646-4cd1-4676-a3d2-595b19e26877 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:46.123044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313ef646-4cd1-4676-a3d2-595b19e26877 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:46.123323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313ef646-4cd1-4676-a3d2-595b19e26877 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:46.123592 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313ef646-4cd1-4676-a3d2-595b19e26877 None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:46.124276 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313ef646-4cd1-4676-a3d2-595b19e26877 None admin] RBAC: Adding query filter params (type=network) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:46.124573 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313ef646-4cd1-4676-a3d2-595b19e26877 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:46.125935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313ef646-4cd1-4676-a3d2-595b19e26877 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:46.135848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 264/529] 10.222.0.16 () {56 vars in 1112 bytes} [Sun Oct 30 00:43:45 2022] GET /identity/v3/services?type=network => generated 124 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:47.044353 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5b0760-82b9-409a-a890-fbd4e864e227 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:47.044919 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5b0760-82b9-409a-a890-fbd4e864e227 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:47.044919 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5b0760-82b9-409a-a890-fbd4e864e227 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:47.046544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 266/530] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:47.059798 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9357e7-f432-4d96-9140-54cf1ca162c0 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:47.060311 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9357e7-f432-4d96-9140-54cf1ca162c0 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:47.060406 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9357e7-f432-4d96-9140-54cf1ca162c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:47.084443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-8f9357e7-f432-4d96-9140-54cf1ca162c0 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:47.085628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-8f9357e7-f432-4d96-9140-54cf1ca162c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:47.142676 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:47.176961 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 265/531] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:47.627031 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5b0760-82b9-409a-a890-fbd4e864e227 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:47.762249 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5b0760-82b9-409a-a890-fbd4e864e227 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:47.775512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:47.809309 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2301923f-f672-4a4a-8f0a-51ef6df2eb51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:47.809945 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2301923f-f672-4a4a-8f0a-51ef6df2eb51 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:47.810219 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2301923f-f672-4a4a-8f0a-51ef6df2eb51 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:47.810488 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2301923f-f672-4a4a-8f0a-51ef6df2eb51 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:47.827957 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2301923f-f672-4a4a-8f0a-51ef6df2eb51 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:47.832760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-2301923f-f672-4a4a-8f0a-51ef6df2eb51 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:43:47.832760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:47.832760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:47.832760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:47.832760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:47.832760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:47.832760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:47.832760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:47.832760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:47.832760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:47.832760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:47.832760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:47.832760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:47.832760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:47.832760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:47.832760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:47.833663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:47.833663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:47.833663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:47.833663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:47.833663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:47.833663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:47.833663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:47.833663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:47.833663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:47.833663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:47.833663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:47.833663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:47.833663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:47.833663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:47.833663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:47.833663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:47.835096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:47.835096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:47.835096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:47.835096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:47.835096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:47.835096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:47.835096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:47.835096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:47.835096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:47.835096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:47.835096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:43:47.835096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:47.835096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 267/532] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:43:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:47.839972 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9357e7-f432-4d96-9140-54cf1ca162c0 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:48.027004 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9357e7-f432-4d96-9140-54cf1ca162c0 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:48.078798 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5294e1-bbe0-4598-bf70-277abd8a0220 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:48.079682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5294e1-bbe0-4598-bf70-277abd8a0220 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:48.079990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5294e1-bbe0-4598-bf70-277abd8a0220 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:48.080298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5294e1-bbe0-4598-bf70-277abd8a0220 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:48.080962 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5294e1-bbe0-4598-bf70-277abd8a0220 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:48.081370 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5294e1-bbe0-4598-bf70-277abd8a0220 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:48.082741 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5294e1-bbe0-4598-bf70-277abd8a0220 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:48.095443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 266/533] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:43:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:48.100762 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2301923f-f672-4a4a-8f0a-51ef6df2eb51 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:48.108584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4959f16-ff03-4cdd-bae8-4e166ab3a8ea None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:48.108865 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4959f16-ff03-4cdd-bae8-4e166ab3a8ea None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:48.109137 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4959f16-ff03-4cdd-bae8-4e166ab3a8ea None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:48.110768 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 267/534] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:43:48.115893 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae45b72-9a48-4d87-ac27-87ffc45daffd None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:48.116170 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae45b72-9a48-4d87-ac27-87ffc45daffd None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:48.116437 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae45b72-9a48-4d87-ac27-87ffc45daffd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:48.143947 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-7ae45b72-9a48-4d87-ac27-87ffc45daffd None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:48.146439 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-7ae45b72-9a48-4d87-ac27-87ffc45daffd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:48.200244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:48.252300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2301923f-f672-4a4a-8f0a-51ef6df2eb51 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:48.266651 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 268/535] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 30 00:43:48.287245 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23c1b6e-074e-454e-b266-fc277e717cec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:48.287824 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23c1b6e-074e-454e-b266-fc277e717cec None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:48.288101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23c1b6e-074e-454e-b266-fc277e717cec None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:48.288371 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23c1b6e-074e-454e-b266-fc277e717cec None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:48.299458 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23c1b6e-074e-454e-b266-fc277e717cec None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:48.301025 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23c1b6e-074e-454e-b266-fc277e717cec None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:48.316130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-f23c1b6e-074e-454e-b266-fc277e717cec None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:48.316130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:48.316130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:48.316130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:48.316130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:48.316130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:48.316130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:48.316130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:48.316130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:48.316130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:48.316130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:48.316130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:48.316130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:48.316130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:48.316130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:48.316130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:48.317101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:48.317101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:48.317101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:48.317101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:48.317101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:48.317101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:48.317101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:48.317101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:48.317101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:48.317101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:48.317101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:48.317101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:48.317101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:48.317101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:48.317101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:48.317101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:48.317101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:48.318568 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:48.318568 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:48.318568 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:48.318568 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 268/536] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:48.323067 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae45b72-9a48-4d87-ac27-87ffc45daffd None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:48.462349 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae45b72-9a48-4d87-ac27-87ffc45daffd None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:48.503349 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb645c2c-3ee5-45e7-9ba1-57bcf62efff6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:48.503985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb645c2c-3ee5-45e7-9ba1-57bcf62efff6 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:48.504260 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb645c2c-3ee5-45e7-9ba1-57bcf62efff6 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:48.504528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb645c2c-3ee5-45e7-9ba1-57bcf62efff6 None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:48.505166 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb645c2c-3ee5-45e7-9ba1-57bcf62efff6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:48.505460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb645c2c-3ee5-45e7-9ba1-57bcf62efff6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:48.506826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb645c2c-3ee5-45e7-9ba1-57bcf62efff6 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:48.518746 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 269/537] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:48.524836 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23c1b6e-074e-454e-b266-fc277e717cec None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:48.572299 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:48.577765 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb645c2c-3ee5-45e7-9ba1-57bcf62efff6 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:48.638801 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:48.667642 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23c1b6e-074e-454e-b266-fc277e717cec None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:48.708410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b290f032-7077-4ed1-bbc6-a97162cc7898 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:48.709056 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b290f032-7077-4ed1-bbc6-a97162cc7898 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:48.709324 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b290f032-7077-4ed1-bbc6-a97162cc7898 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:48.709587 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b290f032-7077-4ed1-bbc6-a97162cc7898 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:48.718516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b290f032-7077-4ed1-bbc6-a97162cc7898 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:48.720052 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b290f032-7077-4ed1-bbc6-a97162cc7898 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:48.732648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-b290f032-7077-4ed1-bbc6-a97162cc7898 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 30 00:43:48.732648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:48.732648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:48.732648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:48.732648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:48.732648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:48.732648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:48.732648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:48.732648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:48.732648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:48.732648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:48.732648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:48.732648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:43:48.732648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:43:48.732648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:43:48.732648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:43:48.733632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:48.733632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:48.733632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:43:48.733632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:48.733632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:43:48.733632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:48.733632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:48.733632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:48.733632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:48.733632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:48.733632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:48.733632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:48.733632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:48.733632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:48.733632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:48.733632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:48.733632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:48.733632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:48.734585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:48.734585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:48.734585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:43:48.734585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:43:48.734585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:43:48.734585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:43:48.734585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:43:48.734585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:43:48.734585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 30 00:43:48.734585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:48.735109 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb645c2c-3ee5-45e7-9ba1-57bcf62efff6 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:48.735193 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 269/538] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:43:48 2022] GET /identity/v3/users/swift => generated 83 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:48.735818 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0836aef8-1436-4f02-b188-361848b1fbe9 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:48.735818 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0836aef8-1436-4f02-b188-361848b1fbe9 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:48.735994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0836aef8-1436-4f02-b188-361848b1fbe9 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:48.741507 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 270/539] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:48.742321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0836aef8-1436-4f02-b188-361848b1fbe9 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:48.772571 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd67a56-5bdd-4a58-90d6-431a7752584c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:48.773308 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd67a56-5bdd-4a58-90d6-431a7752584c None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:48.773583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd67a56-5bdd-4a58-90d6-431a7752584c None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:48.773955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd67a56-5bdd-4a58-90d6-431a7752584c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:48.786567 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd67a56-5bdd-4a58-90d6-431a7752584c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:48.792984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-9fd67a56-5bdd-4a58-90d6-431a7752584c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:43:48.792984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:48.792984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:48.792984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:48.792984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:48.792984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:48.792984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:48.792984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:48.792984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:48.792984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:48.792984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:48.792984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:48.792984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:48.792984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:48.792984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:48.792984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:48.793914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:48.793914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:48.793914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:48.793914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:48.793914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:48.793914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:48.793914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:48.793914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:48.793914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:48.793914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:48.793914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:48.793914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:48.793914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:48.793914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:48.793914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:48.793914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:48.799751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:48.799751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:48.799751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:48.799751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:48.799751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:48.799751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:48.799751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:48.799751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:48.799751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:48.799751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:48.799751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:43:48.799751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:48.799751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 270/540] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:43:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:43:48.799751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b1681f-986c-432f-ab43-10f2941de085 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:48.800529 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b1681f-986c-432f-ab43-10f2941de085 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:48.800529 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b1681f-986c-432f-ab43-10f2941de085 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:48.817271 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-95b1681f-986c-432f-ab43-10f2941de085 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:48.818436 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-95b1681f-986c-432f-ab43-10f2941de085 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:48.829836 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:48.846551 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0836aef8-1436-4f02-b188-361848b1fbe9 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:48.879871 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f86d307-6878-4633-bc0d-4119c01b3b96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:48.880450 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f86d307-6878-4633-bc0d-4119c01b3b96 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:48.880722 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f86d307-6878-4633-bc0d-4119c01b3b96 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:48.880988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f86d307-6878-4633-bc0d-4119c01b3b96 None admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:48.881758 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f86d307-6878-4633-bc0d-4119c01b3b96 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:48.882054 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f86d307-6878-4633-bc0d-4119c01b3b96 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:48.883775 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f86d307-6878-4633-bc0d-4119c01b3b96 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:48.883883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 271/541] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:48.884883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8de468f-ed5f-423d-aac7-45493381241b None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:48.885346 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8de468f-ed5f-423d-aac7-45493381241b None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:48.885620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8de468f-ed5f-423d-aac7-45493381241b None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:48.887763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 272/542] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:48.892507 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8de468f-ed5f-423d-aac7-45493381241b None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:48.902918 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 271/543] 10.222.0.16 () {56 vars in 1135 bytes} [Sun Oct 30 00:43:48 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:48.904243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d536f25-1be6-426e-bba8-7c818fc4e96d None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:48.904526 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d536f25-1be6-426e-bba8-7c818fc4e96d None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:48.904794 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d536f25-1be6-426e-bba8-7c818fc4e96d None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:48.910161 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 272/544] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:48.910161 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed0df23-f9df-4477-a58f-4fc2de70f181 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:48.910161 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed0df23-f9df-4477-a58f-4fc2de70f181 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:48.910161 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed0df23-f9df-4477-a58f-4fc2de70f181 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:48.919422 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:48.926726 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-3ed0df23-f9df-4477-a58f-4fc2de70f181 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:48.927943 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-3ed0df23-f9df-4477-a58f-4fc2de70f181 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:48.996344 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 273/545] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:48.996974 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed0df23-f9df-4477-a58f-4fc2de70f181 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:49.023523 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8de468f-ed5f-423d-aac7-45493381241b None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:49.061604 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e0442a-717e-472e-9de3-30e4a934ad38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:49.062207 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e0442a-717e-472e-9de3-30e4a934ad38 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:49.062483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e0442a-717e-472e-9de3-30e4a934ad38 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:49.062752 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e0442a-717e-472e-9de3-30e4a934ad38 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:49.063401 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e0442a-717e-472e-9de3-30e4a934ad38 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:49.063699 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e0442a-717e-472e-9de3-30e4a934ad38 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:49.066702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e0442a-717e-472e-9de3-30e4a934ad38 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:49.075312 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 273/546] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:43:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:49.076589 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b50c5d1-0435-48b9-8eb7-59fb4b0e8724 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:49.076869 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b50c5d1-0435-48b9-8eb7-59fb4b0e8724 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:49.077140 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b50c5d1-0435-48b9-8eb7-59fb4b0e8724 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:49.096710 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-8b50c5d1-0435-48b9-8eb7-59fb4b0e8724 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:49.097814 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-8b50c5d1-0435-48b9-8eb7-59fb4b0e8724 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:49.099298 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:49.138869 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed0df23-f9df-4477-a58f-4fc2de70f181 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:49.168968 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 274/547] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 30 00:43:49.169809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b50c5d1-0435-48b9-8eb7-59fb4b0e8724 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:49.181918 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:49.186877 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96378d8-6ad7-4a51-a2aa-9660ed066050 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:49.188437 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96378d8-6ad7-4a51-a2aa-9660ed066050 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:49.189968 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96378d8-6ad7-4a51-a2aa-9660ed066050 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:49.190244 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96378d8-6ad7-4a51-a2aa-9660ed066050 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:49.204178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96378d8-6ad7-4a51-a2aa-9660ed066050 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:49.207065 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96378d8-6ad7-4a51-a2aa-9660ed066050 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:49.226994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-c96378d8-6ad7-4a51-a2aa-9660ed066050 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:49.226994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:49.226994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:49.226994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:49.226994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:49.226994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:49.226994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:49.226994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:49.226994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:49.226994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:49.226994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:49.226994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:49.226994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:49.226994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:49.226994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:49.226994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:49.228001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:49.228001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:49.228001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:49.228001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:49.228001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:49.228001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:49.228001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:49.228001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:49.228001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:49.228001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:49.228001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:49.228001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:49.228001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:49.228001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:49.228001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:49.228001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:49.228001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:49.228888 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:49.228888 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:49.228888 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:49.232088 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 274/548] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:49.233118 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96378d8-6ad7-4a51-a2aa-9660ed066050 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:49.285909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b50c5d1-0435-48b9-8eb7-59fb4b0e8724 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:49.321431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f637bc-32ef-47be-9e8e-8478077b1899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:49.322055 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f637bc-32ef-47be-9e8e-8478077b1899 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:49.322329 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f637bc-32ef-47be-9e8e-8478077b1899 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:49.322595 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f637bc-32ef-47be-9e8e-8478077b1899 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:49.333037 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f637bc-32ef-47be-9e8e-8478077b1899 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:49.334476 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f637bc-32ef-47be-9e8e-8478077b1899 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:49.350537 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96378d8-6ad7-4a51-a2aa-9660ed066050 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:49.351693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-f2f637bc-32ef-47be-9e8e-8478077b1899 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:49.351693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:49.351693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:49.351693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:49.351693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:49.351693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:49.351693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:49.351693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:49.351693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:49.351693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:49.351693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:49.351693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:49.351693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:49.351693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:49.351693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:49.351693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:49.352628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:49.352628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:49.352628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:49.352628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:49.352628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:49.352628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:49.352628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:49.352628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:49.352628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:49.352628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:49.352628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:49.352628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:49.352628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:49.352628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:49.352628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:49.352628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:49.352628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:49.353587 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:49.353587 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:49.353587 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:49.354183 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 275/549] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:49.355021 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f637bc-32ef-47be-9e8e-8478077b1899 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:49.378700 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:49.390985 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a45b67-e57b-47cc-bdb3-c77087b56b14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:49.392387 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a45b67-e57b-47cc-bdb3-c77087b56b14 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:49.392387 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a45b67-e57b-47cc-bdb3-c77087b56b14 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:49.392387 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a45b67-e57b-47cc-bdb3-c77087b56b14 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:49.403067 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a45b67-e57b-47cc-bdb3-c77087b56b14 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:49.408565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-55a45b67-e57b-47cc-bdb3-c77087b56b14 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:49.408565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:49.408565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:49.408565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:49.408565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:49.408565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:49.408565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:49.408565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:49.408565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:49.408565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:49.408565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:49.408565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:49.408565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:49.408565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:49.408565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:49.408565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:49.409518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:49.409518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:49.409518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:49.409518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:49.409518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:49.409518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:49.409518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:49.409518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:49.409518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:49.409518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:49.409518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:49.409518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:49.409518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:49.409518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:49.409518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:49.409518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:49.410355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:49.410355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:49.410355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:49.410355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:49.410355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:49.410355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:49.410355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:49.410355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:49.410355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:49.410355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:49.410355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:49.410355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:49.410355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 275/550] 10.222.0.16 () {56 vars in 1102 bytes} [Sun Oct 30 00:43:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:49.411081 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a45b67-e57b-47cc-bdb3-c77087b56b14 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:49.422573 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:49.464543 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f637bc-32ef-47be-9e8e-8478077b1899 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:49.498657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a4a0bc-f95e-4238-b8bf-7f6c54ded4c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:49.499259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a4a0bc-f95e-4238-b8bf-7f6c54ded4c0 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:49.499537 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a4a0bc-f95e-4238-b8bf-7f6c54ded4c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:49.499835 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a4a0bc-f95e-4238-b8bf-7f6c54ded4c0 None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:49.500469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a4a0bc-f95e-4238-b8bf-7f6c54ded4c0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:49.500770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a4a0bc-f95e-4238-b8bf-7f6c54ded4c0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:49.502179 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a4a0bc-f95e-4238-b8bf-7f6c54ded4c0 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:49.510229 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a45b67-e57b-47cc-bdb3-c77087b56b14 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:49.513672 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 276/551] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:49.514481 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a4a0bc-f95e-4238-b8bf-7f6c54ded4c0 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:49.543433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c65daf0-59f8-434d-8d25-08d42e107f31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:49.544068 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c65daf0-59f8-434d-8d25-08d42e107f31 None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:49.544343 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c65daf0-59f8-434d-8d25-08d42e107f31 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:49.544632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c65daf0-59f8-434d-8d25-08d42e107f31 None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:49.545284 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c65daf0-59f8-434d-8d25-08d42e107f31 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:49.546655 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c65daf0-59f8-434d-8d25-08d42e107f31 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:49.564169 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 276/552] 10.222.0.16 () {60 vars in 1141 bytes} [Sun Oct 30 00:43:49 2022] POST /identity/v3/services => generated 251 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:43:49.565354 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c65daf0-59f8-434d-8d25-08d42e107f31 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:49.616526 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:49.635615 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a4a0bc-f95e-4238-b8bf-7f6c54ded4c0 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:49.662635 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:49.689838 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc64c69d-5689-4446-9cd3-b7709f8a93eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:49.690236 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c65daf0-59f8-434d-8d25-08d42e107f31 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:49.690551 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc64c69d-5689-4446-9cd3-b7709f8a93eb None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:49.690826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc64c69d-5689-4446-9cd3-b7709f8a93eb None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:49.691090 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc64c69d-5689-4446-9cd3-b7709f8a93eb None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:49.691941 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc64c69d-5689-4446-9cd3-b7709f8a93eb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:49.692088 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc64c69d-5689-4446-9cd3-b7709f8a93eb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:49.693468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc64c69d-5689-4446-9cd3-b7709f8a93eb None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:49.707068 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 277/553] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:49.707958 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc64c69d-5689-4446-9cd3-b7709f8a93eb None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:49.733315 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba884e33-74c5-4f99-96c5-986c516fec50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:49.734024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba884e33-74c5-4f99-96c5-986c516fec50 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:49.734295 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba884e33-74c5-4f99-96c5-986c516fec50 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:49.734561 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba884e33-74c5-4f99-96c5-986c516fec50 None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:49.735229 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba884e33-74c5-4f99-96c5-986c516fec50 None admin] RBAC: Adding query filter params (name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:49.735525 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba884e33-74c5-4f99-96c5-986c516fec50 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:49.737388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba884e33-74c5-4f99-96c5-986c516fec50 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:49.748677 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 277/554] 10.222.0.16 () {56 vars in 1103 bytes} [Sun Oct 30 00:43:49 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:49.748865 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba884e33-74c5-4f99-96c5-986c516fec50 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:49.822475 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc64c69d-5689-4446-9cd3-b7709f8a93eb None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:49.853161 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba884e33-74c5-4f99-96c5-986c516fec50 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:49.855098 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:49.860474 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c616897-461a-4f92-8231-80b29f372533 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:49.861572 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c616897-461a-4f92-8231-80b29f372533 None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:49.861842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c616897-461a-4f92-8231-80b29f372533 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:49.862108 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c616897-461a-4f92-8231-80b29f372533 None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:49.862753 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c616897-461a-4f92-8231-80b29f372533 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:49.864551 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c616897-461a-4f92-8231-80b29f372533 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:49.883368 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 278/555] 10.222.0.16 () {60 vars in 1141 bytes} [Sun Oct 30 00:43:49 2022] POST /identity/v3/services => generated 240 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:43:49.884274 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c616897-461a-4f92-8231-80b29f372533 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:49.899981 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7f805f-e402-4982-b646-26c8c76ced46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:49.901444 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7f805f-e402-4982-b646-26c8c76ced46 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:49.901722 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7f805f-e402-4982-b646-26c8c76ced46 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:49.903991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7f805f-e402-4982-b646-26c8c76ced46 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:49.916033 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7f805f-e402-4982-b646-26c8c76ced46 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:49.917582 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7f805f-e402-4982-b646-26c8c76ced46 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:49.930976 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-2b7f805f-e402-4982-b646-26c8c76ced46 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:49.930976 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:49.930976 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:49.930976 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:49.930976 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:49.930976 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:49.930976 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:49.930976 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:49.930976 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:49.930976 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:49.930976 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:49.930976 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:49.930976 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:43:49.930976 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:43:49.930976 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:43:49.931991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:43:49.931991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:49.931991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:49.931991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:49.931991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:49.931991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:49.931991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:49.931991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:49.931991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:49.931991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:49.931991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:49.931991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:49.931991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:49.931991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:49.931991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:49.931991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:49.931991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:49.932906 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:43:49.932906 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:43:49.932906 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:43:49.932906 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:43:49.932906 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:43:49.932906 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:43:49.932906 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:49.932906 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:49.936462 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 278/556] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:43:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:49.937493 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7f805f-e402-4982-b646-26c8c76ced46 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:49.947265 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:50.028356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c616897-461a-4f92-8231-80b29f372533 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:50.054432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7f805f-e402-4982-b646-26c8c76ced46 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:50.071298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1bc6203-d459-4b7c-b444-a709c3693d41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:50.072599 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1bc6203-d459-4b7c-b444-a709c3693d41 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:50.072872 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1bc6203-d459-4b7c-b444-a709c3693d41 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:50.073142 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1bc6203-d459-4b7c-b444-a709c3693d41 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:50.081361 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1bc6203-d459-4b7c-b444-a709c3693d41 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:50.083315 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1bc6203-d459-4b7c-b444-a709c3693d41 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:50.094200 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6727175-4e75-42e9-8e5e-8bbdc90129ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:50.095033 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6727175-4e75-42e9-8e5e-8bbdc90129ae None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:50.095312 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6727175-4e75-42e9-8e5e-8bbdc90129ae None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:50.095581 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6727175-4e75-42e9-8e5e-8bbdc90129ae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:50.099216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-e1bc6203-d459-4b7c-b444-a709c3693d41 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 30 00:43:50.099216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:50.099216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:50.099216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:50.099216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:50.099216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:50.099216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:50.099216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:50.099216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:50.099216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:50.099216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:50.099216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:50.099216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:43:50.099216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:43:50.099216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:43:50.099216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:43:50.100469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:50.100469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:50.100469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:43:50.100469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:50.100469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:43:50.100469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:50.100469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:50.100469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:50.100469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:50.100469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:50.100469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:50.100469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:50.100469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:50.100469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:50.100469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:50.100469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:50.100469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:50.100469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:50.102268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:50.102268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:50.102268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:43:50.102268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:43:50.102268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:43:50.102268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:43:50.102268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:43:50.102268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:43:50.102268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 30 00:43:50.102268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:50.102268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 279/557] 10.222.0.16 () {56 vars in 1093 bytes} [Sun Oct 30 00:43:49 2022] GET /identity/v3/users/nova => generated 82 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:50.102268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1bc6203-d459-4b7c-b444-a709c3693d41 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:50.114895 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6727175-4e75-42e9-8e5e-8bbdc90129ae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:50.116746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6727175-4e75-42e9-8e5e-8bbdc90129ae None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:50.132420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-e6727175-4e75-42e9-8e5e-8bbdc90129ae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:50.132420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:50.132420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:50.132420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:50.132420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:50.132420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:50.132420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:50.132420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:50.132420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:50.132420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:50.132420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:50.132420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:50.132420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:50.132420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:50.132420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:50.132420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:50.133535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:50.133535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:50.133535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:50.133535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:50.133535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:50.133535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:50.133535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:50.133535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:50.133535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:50.133535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:50.133535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:50.133535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:50.133535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:50.133535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:50.133535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:50.133535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:50.133535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:50.134417 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:50.134417 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:50.134417 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:50.134417 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 279/558] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:50.134864 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6727175-4e75-42e9-8e5e-8bbdc90129ae None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:50.148191 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:50.213752 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1bc6203-d459-4b7c-b444-a709c3693d41 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:50.228160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:50.253330 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45ba08a-a3bc-4db7-90b8-1675ce762bdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:50.253935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45ba08a-a3bc-4db7-90b8-1675ce762bdf None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:50.254205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45ba08a-a3bc-4db7-90b8-1675ce762bdf None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:50.254465 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45ba08a-a3bc-4db7-90b8-1675ce762bdf None admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:50.255152 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45ba08a-a3bc-4db7-90b8-1675ce762bdf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:50.255283 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6727175-4e75-42e9-8e5e-8bbdc90129ae None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:50.255448 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45ba08a-a3bc-4db7-90b8-1675ce762bdf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:50.257012 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45ba08a-a3bc-4db7-90b8-1675ce762bdf None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:50.270240 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 280/559] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:43:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:50.271138 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45ba08a-a3bc-4db7-90b8-1675ce762bdf None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:50.292496 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b7e22e-67df-4ab9-8729-dee85ec0d6b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:50.293964 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b7e22e-67df-4ab9-8729-dee85ec0d6b7 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:50.294712 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b7e22e-67df-4ab9-8729-dee85ec0d6b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:50.295201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b7e22e-67df-4ab9-8729-dee85ec0d6b7 None admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:50.296800 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b7e22e-67df-4ab9-8729-dee85ec0d6b7 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:50.297623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b7e22e-67df-4ab9-8729-dee85ec0d6b7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:50.300155 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b7e22e-67df-4ab9-8729-dee85ec0d6b7 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:50.324982 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 280/560] 10.222.0.16 () {56 vars in 1133 bytes} [Sun Oct 30 00:43:50 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:50.326755 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b7e22e-67df-4ab9-8729-dee85ec0d6b7 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:50.377231 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45ba08a-a3bc-4db7-90b8-1675ce762bdf None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:50.413160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96b917c-b59c-480d-b40f-b7e694c8f394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:50.413677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96b917c-b59c-480d-b40f-b7e694c8f394 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:50.413947 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96b917c-b59c-480d-b40f-b7e694c8f394 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:50.414226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96b917c-b59c-480d-b40f-b7e694c8f394 None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:50.414843 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96b917c-b59c-480d-b40f-b7e694c8f394 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:50.415137 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96b917c-b59c-480d-b40f-b7e694c8f394 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:50.416532 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96b917c-b59c-480d-b40f-b7e694c8f394 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:50.426871 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:50.428378 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 281/561] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:50.433194 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96b917c-b59c-480d-b40f-b7e694c8f394 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:50.456276 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:50.472117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b7e22e-67df-4ab9-8729-dee85ec0d6b7 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:50.511516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faf7bff-18e6-4467-9d4e-c920a01f14da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:50.512439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faf7bff-18e6-4467-9d4e-c920a01f14da None admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:50.512875 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faf7bff-18e6-4467-9d4e-c920a01f14da None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:50.513288 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faf7bff-18e6-4467-9d4e-c920a01f14da None admin] PATH_INFO: `/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/67f6eff8c0254cc6a210101dd7af988d/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:50.515280 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9faf7bff-18e6-4467-9d4e-c920a01f14da None admin] RBAC: Authorizing `identity:create_grant(project_id=658799ed31d2472fb36af4aff0f03aab, user_id=67f6eff8c0254cc6a210101dd7af988d, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:50.517169 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9faf7bff-18e6-4467-9d4e-c920a01f14da None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:50.533754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 281/562] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:43:50 2022] PUT /identity/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/67f6eff8c0254cc6a210101dd7af988d/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:43:50.535228 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faf7bff-18e6-4467-9d4e-c920a01f14da None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:50.562155 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96b917c-b59c-480d-b40f-b7e694c8f394 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:50.602098 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad63fa51-ecb7-47dc-ac08-ec910227a617 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:50.603211 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad63fa51-ecb7-47dc-ac08-ec910227a617 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:50.603487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad63fa51-ecb7-47dc-ac08-ec910227a617 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:50.603788 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad63fa51-ecb7-47dc-ac08-ec910227a617 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:50.715035 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faf7bff-18e6-4467-9d4e-c920a01f14da None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:50.750089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad63fa51-ecb7-47dc-ac08-ec910227a617 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:50.751600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad63fa51-ecb7-47dc-ac08-ec910227a617 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:50.774222 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:50.775447 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:50.785279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2542f0d-fdde-4265-9f27-c4a84f383719 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:50.786320 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2542f0d-fdde-4265-9f27-c4a84f383719 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:50.786885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2542f0d-fdde-4265-9f27-c4a84f383719 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:50.787345 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2542f0d-fdde-4265-9f27-c4a84f383719 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:50.795290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2542f0d-fdde-4265-9f27-c4a84f383719 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:50.798127 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2542f0d-fdde-4265-9f27-c4a84f383719 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:50.812731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-e2542f0d-fdde-4265-9f27-c4a84f383719 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 30 00:43:50.812731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:50.812731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:50.812731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:50.812731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:50.812731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:50.812731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:50.812731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:50.812731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:50.812731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:50.812731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:50.812731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:50.812731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:43:50.812731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:43:50.812731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:43:50.812731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:43:50.813735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:50.813735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:50.813735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:43:50.813735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:50.813735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:43:50.813735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:50.813735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:50.813735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:50.813735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:50.813735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:50.813735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:50.813735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:50.813735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:50.813735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:50.813735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:50.813735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:50.813735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:50.813735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:50.814780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:50.814780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:50.814780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:43:50.814780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:43:50.814780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:43:50.814780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:43:50.814780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:43:50.814780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:43:50.814780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 30 00:43:50.814780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:50.817716 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 282/563] 10.222.0.16 () {56 vars in 1117 bytes} [Sun Oct 30 00:43:50 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:50.823475 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2542f0d-fdde-4265-9f27-c4a84f383719 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:50.893835 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 282/564] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:43:50 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 465 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 30 00:43:50.899048 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad63fa51-ecb7-47dc-ac08-ec910227a617 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:50.965421 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2542f0d-fdde-4265-9f27-c4a84f383719 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:51.025189 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2132408b-d0e9-40bc-ae04-a0a9f2eb0760 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:51.027759 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2132408b-d0e9-40bc-ae04-a0a9f2eb0760 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:51.027759 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2132408b-d0e9-40bc-ae04-a0a9f2eb0760 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:51.027759 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2132408b-d0e9-40bc-ae04-a0a9f2eb0760 None admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:51.027759 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2132408b-d0e9-40bc-ae04-a0a9f2eb0760 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:51.027759 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2132408b-d0e9-40bc-ae04-a0a9f2eb0760 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:51.030135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2132408b-d0e9-40bc-ae04-a0a9f2eb0760 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:51.065365 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad63fa51-ecb7-47dc-ac08-ec910227a617 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:51.071147 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 283/565] 10.222.0.16 () {56 vars in 1149 bytes} [Sun Oct 30 00:43:50 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:51.077696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2132408b-d0e9-40bc-ae04-a0a9f2eb0760 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:51.105255 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:51.133266 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6508d7-a07f-4c40-9832-4df67bd913ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:51.135831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6508d7-a07f-4c40-9832-4df67bd913ee None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:51.136963 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6508d7-a07f-4c40-9832-4df67bd913ee None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:51.137235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6508d7-a07f-4c40-9832-4df67bd913ee None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:51.150659 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6508d7-a07f-4c40-9832-4df67bd913ee None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:51.154790 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6508d7-a07f-4c40-9832-4df67bd913ee None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:51.178828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-aa6508d7-a07f-4c40-9832-4df67bd913ee None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:51.178828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:51.178828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:51.178828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:51.178828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:51.178828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:51.178828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:51.178828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:51.178828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:51.178828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:51.178828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:51.178828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:51.178828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:51.178828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:51.178828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:51.178828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:51.179839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:51.179839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:51.179839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:51.179839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:51.179839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:51.179839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:51.179839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:51.179839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:51.179839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:51.179839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:51.179839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:51.179839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:51.179839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:51.179839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:51.179839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:51.179839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:51.179839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:51.181070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:51.181070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:51.181070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:51.181070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 283/566] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:43:51.191653 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6508d7-a07f-4c40-9832-4df67bd913ee None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:51.195282 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:51.301253 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2132408b-d0e9-40bc-ae04-a0a9f2eb0760 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:51.387170 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797ae5d2-68d9-4191-abae-2b55e0d2505d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:51.387840 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797ae5d2-68d9-4191-abae-2b55e0d2505d None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:51.389875 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797ae5d2-68d9-4191-abae-2b55e0d2505d None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:51.390141 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797ae5d2-68d9-4191-abae-2b55e0d2505d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:51.423544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797ae5d2-68d9-4191-abae-2b55e0d2505d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:51.425167 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797ae5d2-68d9-4191-abae-2b55e0d2505d None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:51.429080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6508d7-a07f-4c40-9832-4df67bd913ee None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:51.474860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-797ae5d2-68d9-4191-abae-2b55e0d2505d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:51.474860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:51.474860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:51.474860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:51.474860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:51.474860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:51.474860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:51.474860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:51.474860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:51.474860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:51.474860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:51.474860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:51.474860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:51.474860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:51.474860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:51.474860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:51.475990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:51.475990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:51.475990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:51.475990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:51.475990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:51.475990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:51.475990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:51.475990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:51.475990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:51.475990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:51.475990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:51.475990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:51.475990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:51.475990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:51.475990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:51.475990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:51.475990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:51.476885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:51.476885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:51.476885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:51.488507 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 284/567] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 411 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:51.494528 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797ae5d2-68d9-4191-abae-2b55e0d2505d None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:51.540585 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e09baa-dbe5-4808-b949-c2e2b7621b23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:51.541197 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e09baa-dbe5-4808-b949-c2e2b7621b23 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:51.541469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e09baa-dbe5-4808-b949-c2e2b7621b23 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:51.541734 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e09baa-dbe5-4808-b949-c2e2b7621b23 None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:51.542375 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e09baa-dbe5-4808-b949-c2e2b7621b23 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:51.542670 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e09baa-dbe5-4808-b949-c2e2b7621b23 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:51.554530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e09baa-dbe5-4808-b949-c2e2b7621b23 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:51.584238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 284/568] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 393 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:51.588206 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e09baa-dbe5-4808-b949-c2e2b7621b23 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:51.668291 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:51.732067 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:51.735531 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797ae5d2-68d9-4191-abae-2b55e0d2505d None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:51.795942 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e09baa-dbe5-4808-b949-c2e2b7621b23 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:51.808483 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2de9cc-14f9-45e8-956c-949b70abb627 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:51.809031 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2de9cc-14f9-45e8-956c-949b70abb627 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:51.809303 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2de9cc-14f9-45e8-956c-949b70abb627 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:51.809566 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2de9cc-14f9-45e8-956c-949b70abb627 None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:51.810213 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2de9cc-14f9-45e8-956c-949b70abb627 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:51.810505 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2de9cc-14f9-45e8-956c-949b70abb627 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:51.812301 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2de9cc-14f9-45e8-956c-949b70abb627 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:51.838197 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 285/569] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 344 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:51.843197 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2de9cc-14f9-45e8-956c-949b70abb627 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:51.870631 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600bbc61-16b7-4622-a125-de54f23d4ea4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:51.871268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600bbc61-16b7-4622-a125-de54f23d4ea4 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:51.871542 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600bbc61-16b7-4622-a125-de54f23d4ea4 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:51.871831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600bbc61-16b7-4622-a125-de54f23d4ea4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:51.884645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600bbc61-16b7-4622-a125-de54f23d4ea4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:51.886367 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600bbc61-16b7-4622-a125-de54f23d4ea4 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:51.900239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-600bbc61-16b7-4622-a125-de54f23d4ea4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:51.900239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:51.900239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:51.900239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:51.900239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:51.900239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:51.900239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:51.900239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:51.900239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:51.900239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:51.900239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:51.900239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:51.900239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:43:51.900239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:43:51.900239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:43:51.901225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:43:51.901225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:51.901225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:51.901225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:51.901225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:51.901225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:51.901225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:51.901225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:51.901225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:51.901225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:51.901225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:51.901225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:51.901225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:51.901225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:51.901225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:51.901225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:51.901225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:51.902088 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:43:51.902088 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:43:51.902088 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:43:51.902088 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:43:51.902088 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:43:51.902088 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:43:51.902088 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:51.902088 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:51.907751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 285/570] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:43:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:51.907751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600bbc61-16b7-4622-a125-de54f23d4ea4 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:52.015646 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:52.024680 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600bbc61-16b7-4622-a125-de54f23d4ea4 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:52.066248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1675a552-b72e-4d07-897c-67f48b076c48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:52.066460 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2de9cc-14f9-45e8-956c-949b70abb627 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:52.066772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1675a552-b72e-4d07-897c-67f48b076c48 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:52.067047 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1675a552-b72e-4d07-897c-67f48b076c48 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:52.067312 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1675a552-b72e-4d07-897c-67f48b076c48 None admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:52.068030 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1675a552-b72e-4d07-897c-67f48b076c48 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:52.068497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1675a552-b72e-4d07-897c-67f48b076c48 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:52.070754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1675a552-b72e-4d07-897c-67f48b076c48 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:52.091210 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 286/571] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:43:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:52.092895 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10cd266-f57a-4c26-898f-73a6b367d3f4 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:52.093175 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10cd266-f57a-4c26-898f-73a6b367d3f4 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:52.093442 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10cd266-f57a-4c26-898f-73a6b367d3f4 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:52.095950 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 287/572] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:52.100760 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5553aa7e-667c-400b-9702-3713a811c2b2 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:52.101286 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5553aa7e-667c-400b-9702-3713a811c2b2 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:52.101648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5553aa7e-667c-400b-9702-3713a811c2b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:52.110079 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8fd544b-2238-4fb9-98f7-037f9ac48129 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:52.110744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fd544b-2238-4fb9-98f7-037f9ac48129 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:52.111017 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fd544b-2238-4fb9-98f7-037f9ac48129 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:52.111348 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fd544b-2238-4fb9-98f7-037f9ac48129 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:52.118314 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8fd544b-2238-4fb9-98f7-037f9ac48129 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:52.119888 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8fd544b-2238-4fb9-98f7-037f9ac48129 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:52.121817 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-5553aa7e-667c-400b-9702-3713a811c2b2 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:52.123289 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-5553aa7e-667c-400b-9702-3713a811c2b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:52.131876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-f8fd544b-2238-4fb9-98f7-037f9ac48129 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:52.131876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:52.131876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:52.131876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:52.131876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:52.131876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:52.131876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:52.131876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:52.131876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:52.131876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:52.131876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:52.131876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:52.131876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:43:52.131876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:43:52.131876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:43:52.132773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:43:52.132773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:52.132773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:52.132773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:52.132773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:52.132773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:52.132773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:52.132773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:52.132773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:52.132773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:52.132773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:52.132773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:52.132773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:52.132773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:52.132773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:52.132773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:52.132773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:52.133652 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:43:52.133652 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:43:52.133652 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:43:52.133652 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:43:52.133652 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:43:52.133652 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:43:52.133652 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:52.133652 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:52.137832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 286/573] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:43:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:52.138759 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8fd544b-2238-4fb9-98f7-037f9ac48129 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:52.159607 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:52.204951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 288/574] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:52 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:43:52.206020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5553aa7e-667c-400b-9702-3713a811c2b2 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:52.289871 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8fd544b-2238-4fb9-98f7-037f9ac48129 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:52.302319 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:52.325942 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5553aa7e-667c-400b-9702-3713a811c2b2 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:52.342882 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da14a72d-0a2c-41f6-b86f-20a6b83dca2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:52.343564 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da14a72d-0a2c-41f6-b86f-20a6b83dca2a None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:52.343893 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da14a72d-0a2c-41f6-b86f-20a6b83dca2a None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:52.344161 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da14a72d-0a2c-41f6-b86f-20a6b83dca2a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:52.344961 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da14a72d-0a2c-41f6-b86f-20a6b83dca2a None admin] RBAC: Adding query filter params (role.id=35f59686b5d84291a8d9baf19287eb5a, scope.project.id=658799ed31d2472fb36af4aff0f03aab, user.id=e7a886715b5b4c1d99f0b37c58cb5417) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:52.345343 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da14a72d-0a2c-41f6-b86f-20a6b83dca2a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:52.346925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da14a72d-0a2c-41f6-b86f-20a6b83dca2a None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:52.363033 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 287/575] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:43:52 2022] GET /identity/v3/role_assignments?user.id=e7a886715b5b4c1d99f0b37c58cb5417&scope.project.id=658799ed31d2472fb36af4aff0f03aab&role.id=35f59686b5d84291a8d9baf19287eb5a => generated 608 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:52.397956 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eae49a1-8483-4102-a9af-6a53d45b2977 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:52.398583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eae49a1-8483-4102-a9af-6a53d45b2977 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:52.398858 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eae49a1-8483-4102-a9af-6a53d45b2977 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:52.399126 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eae49a1-8483-4102-a9af-6a53d45b2977 None admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:52.399849 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eae49a1-8483-4102-a9af-6a53d45b2977 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:52.400146 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eae49a1-8483-4102-a9af-6a53d45b2977 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:52.401582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eae49a1-8483-4102-a9af-6a53d45b2977 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:52.420177 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 289/576] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:43:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:52.425280 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da14a72d-0a2c-41f6-b86f-20a6b83dca2a None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:52.476432 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226d9306-7a5c-43f7-b080-ed572f5c9f05 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:52.477082 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226d9306-7a5c-43f7-b080-ed572f5c9f05 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:52.477562 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226d9306-7a5c-43f7-b080-ed572f5c9f05 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:52.479338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 290/577] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:43:52.484895 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb42bd4-6bbe-4a4b-b9a0-7f1ba0ba197c None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:52.485533 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb42bd4-6bbe-4a4b-b9a0-7f1ba0ba197c None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:52.486013 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb42bd4-6bbe-4a4b-b9a0-7f1ba0ba197c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:52.511105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-2fb42bd4-6bbe-4a4b-b9a0-7f1ba0ba197c None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:52.513465 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-2fb42bd4-6bbe-4a4b-b9a0-7f1ba0ba197c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:52.566896 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:52.590059 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da14a72d-0a2c-41f6-b86f-20a6b83dca2a None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:52.623746 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 291/578] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:52 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 30 00:43:52.623746 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb42bd4-6bbe-4a4b-b9a0-7f1ba0ba197c None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:52.648279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866f5f90-3881-4a93-9678-a597f1f02ea7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:52.649326 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866f5f90-3881-4a93-9678-a597f1f02ea7 None admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:52.649602 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866f5f90-3881-4a93-9678-a597f1f02ea7 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:52.649925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866f5f90-3881-4a93-9678-a597f1f02ea7 None admin] PATH_INFO: `/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/4716e7afe2784e2a9b872996d895c374/roles/5a0f6f642a10411da6fc1bfc92a7a3d3` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:52.672005 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866f5f90-3881-4a93-9678-a597f1f02ea7 None admin] RBAC: Authorizing `identity:create_grant(project_id=658799ed31d2472fb36af4aff0f03aab, user_id=4716e7afe2784e2a9b872996d895c374, role_id=5a0f6f642a10411da6fc1bfc92a7a3d3)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:52.674393 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866f5f90-3881-4a93-9678-a597f1f02ea7 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:52.698715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 288/579] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:43:52 2022] PUT /identity/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/4716e7afe2784e2a9b872996d895c374/roles/5a0f6f642a10411da6fc1bfc92a7a3d3 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:43:52.716003 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:52.758477 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb42bd4-6bbe-4a4b-b9a0-7f1ba0ba197c None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:52.797870 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39dbd32a-a791-48b2-92c6-e76073b535be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:52.799074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dbd32a-a791-48b2-92c6-e76073b535be None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:52.799600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dbd32a-a791-48b2-92c6-e76073b535be None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:52.800118 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dbd32a-a791-48b2-92c6-e76073b535be None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:52.800911 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39dbd32a-a791-48b2-92c6-e76073b535be None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:52.802510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39dbd32a-a791-48b2-92c6-e76073b535be None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:52.814848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-39dbd32a-a791-48b2-92c6-e76073b535be None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 30 00:43:52.814848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:52.814848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:43:52.814848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:43:52.814848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:43:52.814848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:43:52.814848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:43:52.814848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:43:52.814848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 30 00:43:52.814848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:52.814848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:43:52.814848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:52.814848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:52.814848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:52.814848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:52.814848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:52.814848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:52.816032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:52.816032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:52.816032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:52.816032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:52.816032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:52.816032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:52.816032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:43:52.816032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:43:52.816032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:43:52.816032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:43:52.816032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:52.816032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:52.816032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:52.816032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:52.816032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:52.816032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:52.816932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:52.816932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:52.816932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:52.816932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:52.816932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:52.816932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:52.816932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:52.816932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:52.816932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:52.816932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:52.816932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:43:52.816932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:43:52.816932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 30 00:43:52.816932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:52.820128 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 292/580] 10.222.0.16 () {56 vars in 1129 bytes} [Sun Oct 30 00:43:52 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:52.828952 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866f5f90-3881-4a93-9678-a597f1f02ea7 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:52.930670 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39dbd32a-a791-48b2-92c6-e76073b535be None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:52.955511 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866f5f90-3881-4a93-9678-a597f1f02ea7 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:52.989673 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:52.999010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd66c1d-1f68-4a32-ba23-b7561118b3e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:52.999539 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd66c1d-1f68-4a32-ba23-b7561118b3e1 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:53.001042 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd66c1d-1f68-4a32-ba23-b7561118b3e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:53.001042 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd66c1d-1f68-4a32-ba23-b7561118b3e1 None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:53.001042 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd66c1d-1f68-4a32-ba23-b7561118b3e1 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:53.001042 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd66c1d-1f68-4a32-ba23-b7561118b3e1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:53.002428 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd66c1d-1f68-4a32-ba23-b7561118b3e1 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:53.009745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 289/581] 10.222.0.16 () {56 vars in 1124 bytes} [Sun Oct 30 00:43:52 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:53.010997 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6492ed-737f-4aa3-ba67-9115a8e8eedf None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:53.011278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6492ed-737f-4aa3-ba67-9115a8e8eedf None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:53.011546 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6492ed-737f-4aa3-ba67-9115a8e8eedf None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:53.013037 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 290/582] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:53.014667 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6492ed-737f-4aa3-ba67-9115a8e8eedf None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:53.055109 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39dbd32a-a791-48b2-92c6-e76073b535be None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:53.097802 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60325130-b45b-42fa-a356-647aac85854c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:53.098734 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60325130-b45b-42fa-a356-647aac85854c None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:53.099010 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60325130-b45b-42fa-a356-647aac85854c None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:53.099273 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60325130-b45b-42fa-a356-647aac85854c None admin] PATH_INFO: `/v3/services/network` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:53.099967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60325130-b45b-42fa-a356-647aac85854c None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:53.101462 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60325130-b45b-42fa-a356-647aac85854c None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:53.103329 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:53.112738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-60325130-b45b-42fa-a356-647aac85854c None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 30 00:43:53.112738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:53.112738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:43:53.112738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:43:53.112738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:43:53.112738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:43:53.112738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:43:53.112738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:43:53.112738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 30 00:43:53.112738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:53.112738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:43:53.112738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:53.112738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:53.112738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:53.112738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:53.112738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:53.112738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:53.113803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:53.113803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:53.113803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:53.113803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:53.113803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:53.113803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:53.113803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:43:53.113803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:43:53.113803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:43:53.113803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:43:53.113803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:53.113803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:53.113803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:53.113803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:53.113803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:53.113803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:53.114700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:53.114700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:53.114700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:53.114700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:53.114700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:53.114700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:53.114700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:53.114700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:53.114700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:53.114700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:53.114700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:43:53.114700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:43:53.114700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 30 00:43:53.114700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:53.115436 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 293/583] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:43:52 2022] GET /identity/v3/services/network => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:43:53.119737 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf63aa4-cc92-4874-afc1-49dc680a03c0 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:53.119737 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf63aa4-cc92-4874-afc1-49dc680a03c0 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:53.119737 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf63aa4-cc92-4874-afc1-49dc680a03c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:53.138878 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-ddf63aa4-cc92-4874-afc1-49dc680a03c0 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:53.140085 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ddf63aa4-cc92-4874-afc1-49dc680a03c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:53.140334 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6492ed-737f-4aa3-ba67-9115a8e8eedf None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:53.179156 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e212f204-51eb-49e3-886f-d3005c53dd80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:53.179833 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e212f204-51eb-49e3-886f-d3005c53dd80 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:53.180088 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e212f204-51eb-49e3-886f-d3005c53dd80 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:53.180391 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e212f204-51eb-49e3-886f-d3005c53dd80 None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:53.181065 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e212f204-51eb-49e3-886f-d3005c53dd80 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:53.181346 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e212f204-51eb-49e3-886f-d3005c53dd80 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:53.182887 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e212f204-51eb-49e3-886f-d3005c53dd80 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:53.193422 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 291/584] 10.222.0.16 () {56 vars in 1124 bytes} [Sun Oct 30 00:43:53 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:53.194408 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e212f204-51eb-49e3-886f-d3005c53dd80 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:53.217916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 294/585] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:53 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:43:53.218791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf63aa4-cc92-4874-afc1-49dc680a03c0 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:53.275951 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:53.327955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e212f204-51eb-49e3-886f-d3005c53dd80 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:53.382738 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140da3ef-3401-4e67-98af-13f92624dad7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:53.384268 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140da3ef-3401-4e67-98af-13f92624dad7 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:53.384945 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140da3ef-3401-4e67-98af-13f92624dad7 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:53.386325 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140da3ef-3401-4e67-98af-13f92624dad7 None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:53.387746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140da3ef-3401-4e67-98af-13f92624dad7 None admin] RBAC: Adding query filter params (name=network) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:53.388831 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140da3ef-3401-4e67-98af-13f92624dad7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:53.391502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140da3ef-3401-4e67-98af-13f92624dad7 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:53.396091 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf63aa4-cc92-4874-afc1-49dc680a03c0 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:53.404427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 292/586] 10.222.0.16 () {56 vars in 1112 bytes} [Sun Oct 30 00:43:53 2022] GET /identity/v3/services?name=network => generated 124 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:53.410185 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140da3ef-3401-4e67-98af-13f92624dad7 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:53.449658 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:53.461621 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bd55bd-6db3-4abd-8156-93ec1c932e71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:53.463114 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bd55bd-6db3-4abd-8156-93ec1c932e71 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:53.464751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bd55bd-6db3-4abd-8156-93ec1c932e71 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:53.465284 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bd55bd-6db3-4abd-8156-93ec1c932e71 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:53.466257 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75bd55bd-6db3-4abd-8156-93ec1c932e71 None admin] RBAC: Adding query filter params (interface=public, service_id=e048e8913a994650a35da19c99dc2027, region_id=RegionOne) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:53.467468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75bd55bd-6db3-4abd-8156-93ec1c932e71 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:53.471272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75bd55bd-6db3-4abd-8156-93ec1c932e71 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:53.485081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 295/587] 10.222.0.16 () {56 vars in 1251 bytes} [Sun Oct 30 00:43:53 2022] GET /identity/v3/endpoints?service_id=e048e8913a994650a35da19c99dc2027&interface=public®ion_id=RegionOne => generated 194 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:53.491518 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bd55bd-6db3-4abd-8156-93ec1c932e71 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:53.575982 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140da3ef-3401-4e67-98af-13f92624dad7 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:53.606379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:53.618911 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba97878-877f-4c6b-b4e3-4666e3f451cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:53.619988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba97878-877f-4c6b-b4e3-4666e3f451cb None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:53.620412 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba97878-877f-4c6b-b4e3-4666e3f451cb None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:53.620865 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba97878-877f-4c6b-b4e3-4666e3f451cb None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:53.621617 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba97878-877f-4c6b-b4e3-4666e3f451cb None admin] RBAC: Adding query filter params (type=network) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:53.622054 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba97878-877f-4c6b-b4e3-4666e3f451cb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:53.622803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bd55bd-6db3-4abd-8156-93ec1c932e71 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:53.623733 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba97878-877f-4c6b-b4e3-4666e3f451cb None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:53.629599 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 293/588] 10.222.0.16 () {56 vars in 1112 bytes} [Sun Oct 30 00:43:53 2022] GET /identity/v3/services?type=network => generated 350 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:53.630819 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba97878-877f-4c6b-b4e3-4666e3f451cb None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:53.661489 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ddd539-bf07-4392-bb8c-737f82851863 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:53.662112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ddd539-bf07-4392-bb8c-737f82851863 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:53.662387 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ddd539-bf07-4392-bb8c-737f82851863 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:53.662654 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ddd539-bf07-4392-bb8c-737f82851863 None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:53.663250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ddd539-bf07-4392-bb8c-737f82851863 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:53.664844 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ddd539-bf07-4392-bb8c-737f82851863 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:53.670657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 296/589] 10.222.0.16 () {56 vars in 1087 bytes} [Sun Oct 30 00:43:53 2022] GET /identity/v3/services => generated 772 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:53.671712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ddd539-bf07-4392-bb8c-737f82851863 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:53.741553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:53.792918 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ddd539-bf07-4392-bb8c-737f82851863 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:53.836845 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a6c6bf-dda9-4e7e-8246-402263723430 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:53.838632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a6c6bf-dda9-4e7e-8246-402263723430 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:53.841239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a6c6bf-dda9-4e7e-8246-402263723430 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:53.841680 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a6c6bf-dda9-4e7e-8246-402263723430 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:53.842558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a6c6bf-dda9-4e7e-8246-402263723430 None admin] RBAC: Adding query filter params (interface=public, service_id=c8d617558ca6461b83358fb92091322b, region_id=RegionOne) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:53.843006 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a6c6bf-dda9-4e7e-8246-402263723430 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:53.844831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a6c6bf-dda9-4e7e-8246-402263723430 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:53.855512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 297/590] 10.222.0.16 () {56 vars in 1251 bytes} [Sun Oct 30 00:43:53 2022] GET /identity/v3/endpoints?service_id=c8d617558ca6461b83358fb92091322b&interface=public®ion_id=RegionOne => generated 194 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:53.863788 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a6c6bf-dda9-4e7e-8246-402263723430 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:53.875745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba97878-877f-4c6b-b4e3-4666e3f451cb None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:53.913340 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f38a85-b46c-442a-a7eb-03308dab2c9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:53.913969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f38a85-b46c-442a-a7eb-03308dab2c9f None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:53.914236 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f38a85-b46c-442a-a7eb-03308dab2c9f None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:53.914499 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f38a85-b46c-442a-a7eb-03308dab2c9f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:53.924927 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f38a85-b46c-442a-a7eb-03308dab2c9f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:53.929626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-a0f38a85-b46c-442a-a7eb-03308dab2c9f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:43:53.929626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:53.929626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:53.929626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:53.929626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:53.929626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:53.929626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:53.929626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:53.929626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:53.929626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:53.929626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:53.929626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:53.929626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:53.929626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:53.929626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:53.929626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:53.930721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:53.930721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:53.930721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:53.930721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:53.930721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:53.930721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:53.930721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:53.930721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:53.930721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:53.930721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:53.930721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:53.930721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:53.930721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:53.930721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:53.930721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:53.930721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:53.931900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:53.931900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:53.931900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:53.931900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:53.931900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:53.931900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:53.931900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:53.931900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:53.931900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:53.931900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:53.931900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:43:53.931900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:53.931900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 294/591] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:43:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 301 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:53.936968 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f38a85-b46c-442a-a7eb-03308dab2c9f None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:53.989877 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a6c6bf-dda9-4e7e-8246-402263723430 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:54.002947 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:54.015660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:54.045308 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf8dc4b-d828-42a1-bea0-935adc6308d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:54.045936 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf8dc4b-d828-42a1-bea0-935adc6308d9 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:54.046221 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf8dc4b-d828-42a1-bea0-935adc6308d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:54.046499 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf8dc4b-d828-42a1-bea0-935adc6308d9 None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:54.047107 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf8dc4b-d828-42a1-bea0-935adc6308d9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:54.048516 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf8dc4b-d828-42a1-bea0-935adc6308d9 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:54.055139 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 298/592] 10.222.0.16 () {56 vars in 1087 bytes} [Sun Oct 30 00:43:53 2022] GET /identity/v3/services => generated 772 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:54.100312 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f38a85-b46c-442a-a7eb-03308dab2c9f None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:54.162120 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c87c890-20d2-4368-9daa-aea17d5deeec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:54.162752 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c87c890-20d2-4368-9daa-aea17d5deeec None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:54.163023 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c87c890-20d2-4368-9daa-aea17d5deeec None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:54.163291 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c87c890-20d2-4368-9daa-aea17d5deeec None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:54.164016 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c87c890-20d2-4368-9daa-aea17d5deeec None admin] RBAC: Adding query filter params (name=admin) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:54.164311 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c87c890-20d2-4368-9daa-aea17d5deeec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:54.165701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c87c890-20d2-4368-9daa-aea17d5deeec None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:54.179248 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 295/593] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:43:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:54.184781 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf8dc4b-d828-42a1-bea0-935adc6308d9 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:54.323975 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf8dc4b-d828-42a1-bea0-935adc6308d9 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:54.377910 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f98b86-fc10-4026-b645-fbe074ebaf10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:54.378556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f98b86-fc10-4026-b645-fbe074ebaf10 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:54.378833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f98b86-fc10-4026-b645-fbe074ebaf10 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:54.379103 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f98b86-fc10-4026-b645-fbe074ebaf10 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:54.394078 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f98b86-fc10-4026-b645-fbe074ebaf10 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:54.395558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f98b86-fc10-4026-b645-fbe074ebaf10 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:54.417742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-55f98b86-fc10-4026-b645-fbe074ebaf10 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:54.417742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:54.417742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:54.417742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:54.417742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:54.417742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:54.417742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:54.417742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:54.417742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:54.417742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:54.417742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:54.417742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:54.417742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:54.417742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:54.417742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:54.417742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:54.420053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:54.420053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:54.420053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:54.420053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:54.420053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:54.420053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:54.420053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:54.420053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:54.420053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:54.420053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:54.420053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:54.420053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:54.420053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:54.420053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:54.420053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:54.420053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:54.420053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:54.422198 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:54.422198 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:54.422198 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:54.422198 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 299/594] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:54.424925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c87c890-20d2-4368-9daa-aea17d5deeec None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:54.551772 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c87c890-20d2-4368-9daa-aea17d5deeec None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:54.566165 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:54.590849 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe8a2bf-97b7-4951-bc73-af23c8da9a55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:54.591522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe8a2bf-97b7-4951-bc73-af23c8da9a55 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:54.591952 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe8a2bf-97b7-4951-bc73-af23c8da9a55 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:54.592309 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe8a2bf-97b7-4951-bc73-af23c8da9a55 None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:54.593049 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe8a2bf-97b7-4951-bc73-af23c8da9a55 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:54.593433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe8a2bf-97b7-4951-bc73-af23c8da9a55 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:54.594999 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe8a2bf-97b7-4951-bc73-af23c8da9a55 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:54.608010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 296/595] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:54.613477 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f98b86-fc10-4026-b645-fbe074ebaf10 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:54.675984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:54.762448 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f98b86-fc10-4026-b645-fbe074ebaf10 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:54.807214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fb6953-e6b7-4dae-a564-253c93ead13c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:54.807891 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fb6953-e6b7-4dae-a564-253c93ead13c None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:54.808165 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fb6953-e6b7-4dae-a564-253c93ead13c None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:54.808432 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fb6953-e6b7-4dae-a564-253c93ead13c None admin] PATH_INFO: `/v3/users/swift` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:54.820028 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6fb6953-e6b7-4dae-a564-253c93ead13c None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:54.821486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6fb6953-e6b7-4dae-a564-253c93ead13c None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:54.839472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-c6fb6953-e6b7-4dae-a564-253c93ead13c None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 30 00:43:54.839472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:54.839472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:54.839472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:54.839472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:54.839472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:54.839472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:54.839472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:54.839472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:54.839472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:54.839472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:54.839472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:54.839472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:43:54.839472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:43:54.839472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:43:54.839472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:43:54.841867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:54.841867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:54.841867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:43:54.841867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:54.841867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:43:54.841867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:54.841867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:54.841867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:54.841867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:54.841867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:54.841867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:54.841867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:54.841867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:54.841867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:54.841867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:54.841867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:54.841867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:54.841867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:54.844244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:54.844244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:54.844244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:43:54.844244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:43:54.844244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:43:54.844244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:43:54.844244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:43:54.844244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:43:54.844244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 30 00:43:54.844244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:54.844244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 300/596] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:43:54 2022] GET /identity/v3/users/swift => generated 83 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:54.847970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe8a2bf-97b7-4951-bc73-af23c8da9a55 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:55.008353 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe8a2bf-97b7-4951-bc73-af23c8da9a55 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:55.052904 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea43f79b-a3cd-4e19-8dc8-c07670238d5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:55.053571 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea43f79b-a3cd-4e19-8dc8-c07670238d5e None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:55.053899 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea43f79b-a3cd-4e19-8dc8-c07670238d5e None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:55.054166 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea43f79b-a3cd-4e19-8dc8-c07670238d5e None admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:55.054899 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea43f79b-a3cd-4e19-8dc8-c07670238d5e None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:55.055192 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea43f79b-a3cd-4e19-8dc8-c07670238d5e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:55.056667 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea43f79b-a3cd-4e19-8dc8-c07670238d5e None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:55.083002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 297/597] 10.222.0.16 () {56 vars in 1135 bytes} [Sun Oct 30 00:43:54 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:55.089055 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fb6953-e6b7-4dae-a564-253c93ead13c None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:55.224162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:55.232015 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fb6953-e6b7-4dae-a564-253c93ead13c None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:55.278428 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3803b1-b1d4-4c3a-973d-10efdf52f7ad None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:55.278625 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f61a9a6-f501-4990-a130-b17101696648 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:55.278756 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3803b1-b1d4-4c3a-973d-10efdf52f7ad None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:55.278971 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3803b1-b1d4-4c3a-973d-10efdf52f7ad None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:55.279064 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f61a9a6-f501-4990-a130-b17101696648 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:55.279334 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f61a9a6-f501-4990-a130-b17101696648 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:55.279603 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f61a9a6-f501-4990-a130-b17101696648 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:55.280666 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 298/598] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:43:55.285677 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1da369e-c7e6-4c18-901b-4b16882a5993 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:55.285954 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1da369e-c7e6-4c18-901b-4b16882a5993 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:55.286218 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1da369e-c7e6-4c18-901b-4b16882a5993 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:55.305345 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-a1da369e-c7e6-4c18-901b-4b16882a5993 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:55.306551 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a1da369e-c7e6-4c18-901b-4b16882a5993 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:55.350254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:55.385010 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f61a9a6-f501-4990-a130-b17101696648 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:55.386861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f61a9a6-f501-4990-a130-b17101696648 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:55.415042 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 299/599] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:55 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 30 00:43:55.417085 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2373dff8-47bd-483c-9349-b42a134b4dfc None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:55.417367 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2373dff8-47bd-483c-9349-b42a134b4dfc None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:55.417639 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2373dff8-47bd-483c-9349-b42a134b4dfc None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:55.423935 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 300/600] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:55.425473 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84ba7c5-f020-4b48-9fc1-02edf5c4cc28 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:55.426028 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84ba7c5-f020-4b48-9fc1-02edf5c4cc28 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:55.426028 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84ba7c5-f020-4b48-9fc1-02edf5c4cc28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:55.447519 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-c84ba7c5-f020-4b48-9fc1-02edf5c4cc28 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:55.448744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-c84ba7c5-f020-4b48-9fc1-02edf5c4cc28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:55.529132 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 301/601] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:55 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:43:55.553987 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 301/602] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:43:55 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 466 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 30 00:43:55.558416 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:55.561106 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84ba7c5-f020-4b48-9fc1-02edf5c4cc28 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:55.663464 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:55.675759 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84ba7c5-f020-4b48-9fc1-02edf5c4cc28 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:55.710391 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480e8187-12ec-4836-b944-15df6bab1399 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:55.711401 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480e8187-12ec-4836-b944-15df6bab1399 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:55.711919 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480e8187-12ec-4836-b944-15df6bab1399 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:55.712402 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480e8187-12ec-4836-b944-15df6bab1399 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:55.723960 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480e8187-12ec-4836-b944-15df6bab1399 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:55.725695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480e8187-12ec-4836-b944-15df6bab1399 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:55.744725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-480e8187-12ec-4836-b944-15df6bab1399 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:55.744725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:55.744725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:55.744725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:55.744725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:55.744725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:55.744725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:55.744725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:55.744725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:55.744725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:55.744725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:55.744725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:55.744725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:55.744725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:55.744725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:55.744725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:55.745685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:55.745685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:55.745685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:55.745685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:55.745685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:55.745685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:55.745685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:55.745685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:55.745685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:55.745685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:55.745685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:55.745685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:55.745685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:55.745685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:55.745685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:55.745685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:55.745685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:55.746587 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:55.746587 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:55.746587 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:55.748340 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 302/603] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:55.754214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f61a9a6-f501-4990-a130-b17101696648 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:55.818968 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480e8187-12ec-4836-b944-15df6bab1399 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:55.897637 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f61a9a6-f501-4990-a130-b17101696648 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:55.938288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f67263c-ab08-4bd8-a63b-d8330ce908c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:55.938922 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f67263c-ab08-4bd8-a63b-d8330ce908c4 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:55.939156 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f67263c-ab08-4bd8-a63b-d8330ce908c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:55.939497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f67263c-ab08-4bd8-a63b-d8330ce908c4 None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:55.940164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f67263c-ab08-4bd8-a63b-d8330ce908c4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:55.940466 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f67263c-ab08-4bd8-a63b-d8330ce908c4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:55.942329 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f67263c-ab08-4bd8-a63b-d8330ce908c4 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:55.947770 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480e8187-12ec-4836-b944-15df6bab1399 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:55.959750 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 302/604] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:55.959750 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f67263c-ab08-4bd8-a63b-d8330ce908c4 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:55.989018 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8bdfca-e9d0-4826-81e0-96377b3b988c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:55.990533 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8bdfca-e9d0-4826-81e0-96377b3b988c None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:55.990983 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8bdfca-e9d0-4826-81e0-96377b3b988c None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:55.991645 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8bdfca-e9d0-4826-81e0-96377b3b988c None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:55.992803 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e8bdfca-e9d0-4826-81e0-96377b3b988c None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:55.994609 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e8bdfca-e9d0-4826-81e0-96377b3b988c None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:56.006631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-3e8bdfca-e9d0-4826-81e0-96377b3b988c None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 30 00:43:56.006631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:56.006631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:43:56.006631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:43:56.006631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:43:56.006631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:43:56.006631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:43:56.006631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:43:56.006631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 30 00:43:56.006631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:56.006631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:43:56.006631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:56.006631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:56.006631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:56.006631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:56.006631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:56.007834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:56.007834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:56.007834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:56.007834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:56.007834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:56.007834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:56.007834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:56.007834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:43:56.007834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:43:56.007834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:43:56.007834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:43:56.007834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:56.007834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:56.007834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:56.007834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:56.007834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:56.007834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:56.008944 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:56.008944 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:56.008944 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:56.008944 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:56.008944 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:56.008944 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:56.008944 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:56.008944 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:56.008944 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:56.008944 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:56.008944 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:43:56.008944 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:43:56.008944 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 30 00:43:56.008944 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:56.017050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 303/605] 10.222.0.16 () {56 vars in 1132 bytes} [Sun Oct 30 00:43:55 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:43:56.018289 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8bdfca-e9d0-4826-81e0-96377b3b988c None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:56.024869 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:56.027335 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:56.095925 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f67263c-ab08-4bd8-a63b-d8330ce908c4 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:56.136191 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8bdfca-e9d0-4826-81e0-96377b3b988c None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:56.142171 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995175b1-f19d-4710-a395-43dc0ec19471 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:56.143206 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995175b1-f19d-4710-a395-43dc0ec19471 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:56.143633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995175b1-f19d-4710-a395-43dc0ec19471 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:56.144613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995175b1-f19d-4710-a395-43dc0ec19471 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:56.159063 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995175b1-f19d-4710-a395-43dc0ec19471 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:43:56.166150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-995175b1-f19d-4710-a395-43dc0ec19471 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:56.166150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:56.166150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:56.166150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:56.166150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:56.166150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:56.166150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:56.166150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:56.166150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:56.166150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:56.166150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:56.166150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:56.166150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:43:56.166150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:43:56.166150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:43:56.166150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:43:56.167080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:43:56.167080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:43:56.167080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:43:56.167080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:43:56.167080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:56.167080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:56.167080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:56.167080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:56.167080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:56.167080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:56.167080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:56.167080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:56.167080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:56.167080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:56.167080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:56.167080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:56.167943 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:56.167943 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:56.167943 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:56.167943 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:56.167943 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:43:56.167943 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:43:56.167943 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:43:56.167943 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:43:56.167943 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:43:56.167943 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:43:56.167943 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:43:56.167943 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:56.171787 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 303/606] 10.222.0.16 () {56 vars in 1102 bytes} [Sun Oct 30 00:43:55 2022] GET /identity/v3/roles/service => generated 85 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:56.172816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995175b1-f19d-4710-a395-43dc0ec19471 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:56.181266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6780980-4283-417f-88f6-74b126952c21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:56.181938 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6780980-4283-417f-88f6-74b126952c21 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:56.182215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6780980-4283-417f-88f6-74b126952c21 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:56.182483 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6780980-4283-417f-88f6-74b126952c21 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:56.190277 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6780980-4283-417f-88f6-74b126952c21 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:56.191763 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6780980-4283-417f-88f6-74b126952c21 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:56.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-c6780980-4283-417f-88f6-74b126952c21 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:56.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:56.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:56.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:56.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:56.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:56.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:56.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:56.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:56.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:56.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:56.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:56.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:43:56.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:43:56.204057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:43:56.205243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:43:56.205243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:56.205243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:56.205243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:56.205243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:56.205243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:56.205243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:56.205243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:56.205243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:56.205243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:56.205243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:56.205243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:56.205243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:56.205243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:56.205243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:56.205243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:56.205243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:56.206513 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:43:56.206513 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:43:56.206513 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:43:56.206513 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:43:56.206513 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:43:56.206513 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:43:56.206513 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:56.206513 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:56.210424 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 304/607] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:43:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:56.211453 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6780980-4283-417f-88f6-74b126952c21 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:56.281596 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:56.322238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995175b1-f19d-4710-a395-43dc0ec19471 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:56.322809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6780980-4283-417f-88f6-74b126952c21 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:56.350275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:56.360428 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76439eb-7e09-45d3-b178-9013ef115f7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:56.365092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76439eb-7e09-45d3-b178-9013ef115f7a None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:56.365092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76439eb-7e09-45d3-b178-9013ef115f7a None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:56.365092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76439eb-7e09-45d3-b178-9013ef115f7a None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:56.365092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76439eb-7e09-45d3-b178-9013ef115f7a None admin] RBAC: Adding query filter params (name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:56.365092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76439eb-7e09-45d3-b178-9013ef115f7a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:56.365491 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76439eb-7e09-45d3-b178-9013ef115f7a None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:56.370108 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5653df40-1def-4497-9567-3efad134926f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:56.370723 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5653df40-1def-4497-9567-3efad134926f None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:56.371006 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5653df40-1def-4497-9567-3efad134926f None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:56.371311 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5653df40-1def-4497-9567-3efad134926f None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:56.371914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5653df40-1def-4497-9567-3efad134926f None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:56.372214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5653df40-1def-4497-9567-3efad134926f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:56.373580 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5653df40-1def-4497-9567-3efad134926f None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:56.379637 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 305/608] 10.222.0.16 () {56 vars in 1103 bytes} [Sun Oct 30 00:43:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:56.379637 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76439eb-7e09-45d3-b178-9013ef115f7a None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:56.379896 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 304/609] 10.222.0.16 () {56 vars in 1126 bytes} [Sun Oct 30 00:43:56 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:56.380443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13cf474-ab49-4f0e-9cfc-b768b0332d79 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:56.380715 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13cf474-ab49-4f0e-9cfc-b768b0332d79 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:56.380982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13cf474-ab49-4f0e-9cfc-b768b0332d79 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:56.382316 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 305/610] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:56.386605 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13cf474-ab49-4f0e-9cfc-b768b0332d79 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:56.489226 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76439eb-7e09-45d3-b178-9013ef115f7a None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:56.491945 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13cf474-ab49-4f0e-9cfc-b768b0332d79 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:56.516603 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:56.527436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ed0a65-9f0b-4861-8479-b9db392b5184 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:56.528428 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ed0a65-9f0b-4861-8479-b9db392b5184 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:56.528554 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ec52da-d1bc-473a-9292-f59b007e1a20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:56.529018 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ed0a65-9f0b-4861-8479-b9db392b5184 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:56.529122 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ec52da-d1bc-473a-9292-f59b007e1a20 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:56.529319 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ec52da-d1bc-473a-9292-f59b007e1a20 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:56.529585 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ec52da-d1bc-473a-9292-f59b007e1a20 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:56.529932 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ed0a65-9f0b-4861-8479-b9db392b5184 None admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:56.530941 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ed0a65-9f0b-4861-8479-b9db392b5184 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:56.531445 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ed0a65-9f0b-4861-8479-b9db392b5184 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:56.533133 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ed0a65-9f0b-4861-8479-b9db392b5184 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:56.540618 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ec52da-d1bc-473a-9292-f59b007e1a20 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:56.542051 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ec52da-d1bc-473a-9292-f59b007e1a20 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:56.546319 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 306/611] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:43:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:56.547310 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ed0a65-9f0b-4861-8479-b9db392b5184 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:56.559955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-33ec52da-d1bc-473a-9292-f59b007e1a20 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:56.559955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:56.559955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:56.559955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:56.559955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:56.559955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:56.559955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:56.559955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:56.559955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:56.559955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:56.559955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:56.559955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:56.559955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:56.559955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:56.559955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:56.559955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:56.561285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:56.561285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:56.561285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:56.561285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:56.561285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:56.561285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:56.561285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:56.561285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:56.561285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:56.561285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:56.561285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:56.561285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:56.561285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:56.561285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:56.561285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:56.561285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:56.561285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:56.562158 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:56.562158 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:56.562158 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:56.562158 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 306/612] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:56.562734 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c77aac7-be53-45a6-96bf-94733ac776e2 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:56.562975 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c77aac7-be53-45a6-96bf-94733ac776e2 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:56.563242 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c77aac7-be53-45a6-96bf-94733ac776e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:56.582190 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-7c77aac7-be53-45a6-96bf-94733ac776e2 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:56.583372 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-7c77aac7-be53-45a6-96bf-94733ac776e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:56.610801 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:56.656912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 307/613] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:56 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:43:56.657758 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c77aac7-be53-45a6-96bf-94733ac776e2 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:56.664933 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ed0a65-9f0b-4861-8479-b9db392b5184 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:56.700885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e97a43-abdc-41b9-9e5d-d88eebd6df7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:56.701538 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e97a43-abdc-41b9-9e5d-d88eebd6df7a None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:56.701815 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e97a43-abdc-41b9-9e5d-d88eebd6df7a None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:56.702083 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e97a43-abdc-41b9-9e5d-d88eebd6df7a None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:56.702708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e97a43-abdc-41b9-9e5d-d88eebd6df7a None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:56.703030 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e97a43-abdc-41b9-9e5d-d88eebd6df7a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:56.704487 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e97a43-abdc-41b9-9e5d-d88eebd6df7a None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:56.715760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 307/614] 10.222.0.16 () {56 vars in 1126 bytes} [Sun Oct 30 00:43:56 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:56.715760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e97a43-abdc-41b9-9e5d-d88eebd6df7a None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:56.797035 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:56.797368 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c77aac7-be53-45a6-96bf-94733ac776e2 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:56.840817 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad202557-fe18-4d16-8185-25a4c6b26353 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:56.841711 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad202557-fe18-4d16-8185-25a4c6b26353 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:56.841986 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad202557-fe18-4d16-8185-25a4c6b26353 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:56.842252 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad202557-fe18-4d16-8185-25a4c6b26353 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:56.843081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad202557-fe18-4d16-8185-25a4c6b26353 None admin] RBAC: Adding query filter params (role.id=35f59686b5d84291a8d9baf19287eb5a, scope.project.id=658799ed31d2472fb36af4aff0f03aab, user.id=67f6eff8c0254cc6a210101dd7af988d) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:56.843387 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad202557-fe18-4d16-8185-25a4c6b26353 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:56.845221 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad202557-fe18-4d16-8185-25a4c6b26353 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:56.853675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 308/615] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:43:56 2022] GET /identity/v3/role_assignments?user.id=67f6eff8c0254cc6a210101dd7af988d&scope.project.id=658799ed31d2472fb36af4aff0f03aab&role.id=35f59686b5d84291a8d9baf19287eb5a => generated 608 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:56.855200 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc4f295-00e2-4532-b439-e9d9bbb32160 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:56.855486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc4f295-00e2-4532-b439-e9d9bbb32160 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:56.855775 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc4f295-00e2-4532-b439-e9d9bbb32160 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:56.857487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 309/616] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:56.866198 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53182149-c551-4ab5-8cc4-de36c915e1a2 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:56.866544 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53182149-c551-4ab5-8cc4-de36c915e1a2 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:56.866910 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53182149-c551-4ab5-8cc4-de36c915e1a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:56.880366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e97a43-abdc-41b9-9e5d-d88eebd6df7a None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:56.889768 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-53182149-c551-4ab5-8cc4-de36c915e1a2 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:56.891132 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-53182149-c551-4ab5-8cc4-de36c915e1a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:56.916311 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:56.933461 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd466cb7-7a1b-4587-910f-d3a5613d89e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:56.934364 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd466cb7-7a1b-4587-910f-d3a5613d89e7 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:56.934785 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd466cb7-7a1b-4587-910f-d3a5613d89e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:56.935197 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd466cb7-7a1b-4587-910f-d3a5613d89e7 None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:56.936341 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd466cb7-7a1b-4587-910f-d3a5613d89e7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:56.936836 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd466cb7-7a1b-4587-910f-d3a5613d89e7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:56.939251 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd466cb7-7a1b-4587-910f-d3a5613d89e7 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:56.954354 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 308/617] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:56.955772 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd466cb7-7a1b-4587-910f-d3a5613d89e7 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:56.977871 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 310/618] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:56 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 30 00:43:56.978745 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53182149-c551-4ab5-8cc4-de36c915e1a2 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:57.087231 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53182149-c551-4ab5-8cc4-de36c915e1a2 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:57.123946 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4289a434-d6ac-40d7-bea1-7caf371f33d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:57.124498 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4289a434-d6ac-40d7-bea1-7caf371f33d2 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:57.124768 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4289a434-d6ac-40d7-bea1-7caf371f33d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:57.125030 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4289a434-d6ac-40d7-bea1-7caf371f33d2 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:57.132645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4289a434-d6ac-40d7-bea1-7caf371f33d2 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:57.134105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4289a434-d6ac-40d7-bea1-7caf371f33d2 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:57.147077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-4289a434-d6ac-40d7-bea1-7caf371f33d2 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 30 00:43:57.147077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:57.147077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:57.147077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:57.147077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:57.147077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:57.147077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:57.147077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:57.147077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:57.147077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:57.147077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:57.147077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:57.147077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:43:57.147077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:43:57.147077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:43:57.147077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:43:57.148112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:57.148112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:57.148112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:43:57.148112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:57.148112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:43:57.148112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:43:57.148112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:57.148112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:57.148112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:57.148112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:57.148112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:57.148112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:57.148112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:57.148112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:57.148112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:57.148112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:57.148112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:57.148112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:57.149272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:57.149272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:57.149272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:43:57.149272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:43:57.149272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:43:57.149272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:43:57.149272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:43:57.149272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:43:57.149272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 30 00:43:57.149272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:57.149272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 311/619] 10.222.0.16 () {56 vars in 1117 bytes} [Sun Oct 30 00:43:56 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:57.154211 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4289a434-d6ac-40d7-bea1-7caf371f33d2 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:57.154894 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd466cb7-7a1b-4587-910f-d3a5613d89e7 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:57.167203 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:57.175514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:57.197984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a967ab99-a10b-45f8-85cc-673f6bb6e186 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:57.199020 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a967ab99-a10b-45f8-85cc-673f6bb6e186 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:57.199490 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a967ab99-a10b-45f8-85cc-673f6bb6e186 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:57.199967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a967ab99-a10b-45f8-85cc-673f6bb6e186 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:57.200803 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a967ab99-a10b-45f8-85cc-673f6bb6e186 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:57.202455 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a967ab99-a10b-45f8-85cc-673f6bb6e186 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:57.212410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-a967ab99-a10b-45f8-85cc-673f6bb6e186 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 30 00:43:57.212410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:57.212410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:43:57.212410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:43:57.212410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:43:57.212410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:43:57.212410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:43:57.212410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:43:57.212410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 30 00:43:57.212410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:57.212410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:43:57.212410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:57.212410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:57.212410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:57.212410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:57.212410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:57.212410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:57.213427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:57.213427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:57.213427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:57.213427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:57.213427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:57.213427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:57.213427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:43:57.213427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:43:57.213427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:43:57.213427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:43:57.213427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:57.213427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:57.213427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:57.213427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:57.213427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:57.213427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:57.214279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:57.214279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:57.214279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:57.214279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:57.214279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:57.214279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:57.214279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:57.214279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:57.214279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:57.214279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:57.214279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:43:57.214279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:43:57.214279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 30 00:43:57.214279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:43:57.216823 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 309/620] 10.222.0.16 () {56 vars in 1129 bytes} [Sun Oct 30 00:43:56 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:57.226023 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a967ab99-a10b-45f8-85cc-673f6bb6e186 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:57.284469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4289a434-d6ac-40d7-bea1-7caf371f33d2 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:57.323313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5cafdc-d4d2-4222-a9f0-f6b96db0692d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:57.324807 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5cafdc-d4d2-4222-a9f0-f6b96db0692d None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:57.325345 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5cafdc-d4d2-4222-a9f0-f6b96db0692d None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:57.325855 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5cafdc-d4d2-4222-a9f0-f6b96db0692d None admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:57.327432 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5cafdc-d4d2-4222-a9f0-f6b96db0692d None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:57.327980 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5cafdc-d4d2-4222-a9f0-f6b96db0692d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:57.330854 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5cafdc-d4d2-4222-a9f0-f6b96db0692d None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:57.356250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 312/621] 10.222.0.16 () {56 vars in 1149 bytes} [Sun Oct 30 00:43:57 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:43:57.356250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5cafdc-d4d2-4222-a9f0-f6b96db0692d None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:57.362309 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a967ab99-a10b-45f8-85cc-673f6bb6e186 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:57.399000 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ec9ed1-6d31-4876-95f9-997cfa85ce10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:57.399522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ec9ed1-6d31-4876-95f9-997cfa85ce10 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:57.399835 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ec9ed1-6d31-4876-95f9-997cfa85ce10 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:57.400103 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ec9ed1-6d31-4876-95f9-997cfa85ce10 None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:57.401234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ec9ed1-6d31-4876-95f9-997cfa85ce10 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:57.401234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ec9ed1-6d31-4876-95f9-997cfa85ce10 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:57.402453 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ec9ed1-6d31-4876-95f9-997cfa85ce10 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:57.411769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 310/622] 10.222.0.16 () {56 vars in 1124 bytes} [Sun Oct 30 00:43:57 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:57.411769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ec9ed1-6d31-4876-95f9-997cfa85ce10 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:57.449490 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:57.451692 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:57.478216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5cafdc-d4d2-4222-a9f0-f6b96db0692d None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:57.513184 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681ba1f0-2f14-4920-b1fc-76f3171616ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:57.514297 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681ba1f0-2f14-4920-b1fc-76f3171616ce None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:57.514406 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ec9ed1-6d31-4876-95f9-997cfa85ce10 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:57.515204 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681ba1f0-2f14-4920-b1fc-76f3171616ce None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:57.515660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681ba1f0-2f14-4920-b1fc-76f3171616ce None admin] PATH_INFO: `/v3/services/network` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:57.516519 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681ba1f0-2f14-4920-b1fc-76f3171616ce None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:57.518501 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681ba1f0-2f14-4920-b1fc-76f3171616ce None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:57.528005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-681ba1f0-2f14-4920-b1fc-76f3171616ce None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 30 00:43:57.528005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:57.528005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:43:57.528005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:43:57.528005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:43:57.528005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:43:57.528005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:43:57.528005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:43:57.528005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 30 00:43:57.528005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:57.528005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:43:57.528005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:57.528005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:57.528005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:57.528005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:57.528005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:57.528005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:57.529034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:57.529034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:57.529034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:57.529034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:57.529034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:57.529034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:57.529034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:43:57.529034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:43:57.529034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:43:57.529034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:43:57.529034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:57.529034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:57.529034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:57.529034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:57.529034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:57.529034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:57.529860 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:57.529860 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:57.529860 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:57.529860 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:57.529860 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:57.529860 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:57.529860 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:57.529860 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:57.529860 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:57.529860 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:57.529860 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:43:57.529860 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:43:57.529860 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 30 00:43:57.529860 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:57.532412 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 313/623] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:43:57 2022] GET /identity/v3/services/network => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:57.533446 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681ba1f0-2f14-4920-b1fc-76f3171616ce None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:57.554066 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2be8ea-f97f-49ac-8b08-6fc3d7c9aa3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:57.554759 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2be8ea-f97f-49ac-8b08-6fc3d7c9aa3d None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:57.555024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2be8ea-f97f-49ac-8b08-6fc3d7c9aa3d None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:57.555289 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2be8ea-f97f-49ac-8b08-6fc3d7c9aa3d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:57.626302 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2be8ea-f97f-49ac-8b08-6fc3d7c9aa3d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:57.627923 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2be8ea-f97f-49ac-8b08-6fc3d7c9aa3d None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:57.637782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681ba1f0-2f14-4920-b1fc-76f3171616ce None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:57.672017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34152e36-8f4b-4cbf-8cac-ca4ebc60853e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:57.672958 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34152e36-8f4b-4cbf-8cac-ca4ebc60853e None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:57.673423 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34152e36-8f4b-4cbf-8cac-ca4ebc60853e None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:57.674125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34152e36-8f4b-4cbf-8cac-ca4ebc60853e None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:57.674941 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34152e36-8f4b-4cbf-8cac-ca4ebc60853e None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:57.675443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34152e36-8f4b-4cbf-8cac-ca4ebc60853e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:57.676066 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:57.677202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34152e36-8f4b-4cbf-8cac-ca4ebc60853e None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:57.687522 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 314/624] 10.222.0.16 () {56 vars in 1124 bytes} [Sun Oct 30 00:43:57 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:57.688585 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34152e36-8f4b-4cbf-8cac-ca4ebc60853e None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:57.691835 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:57.735780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 311/625] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:43:57 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 30 00:43:57.737136 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2be8ea-f97f-49ac-8b08-6fc3d7c9aa3d None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:57.800173 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34152e36-8f4b-4cbf-8cac-ca4ebc60853e None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:57.836468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c588513-f323-484a-9244-6696981412cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:57.837218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c588513-f323-484a-9244-6696981412cd None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:57.837438 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c588513-f323-484a-9244-6696981412cd None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:57.837711 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c588513-f323-484a-9244-6696981412cd None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:57.838337 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c588513-f323-484a-9244-6696981412cd None admin] RBAC: Adding query filter params (name=network) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:57.838894 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c588513-f323-484a-9244-6696981412cd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:57.840121 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c588513-f323-484a-9244-6696981412cd None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:57.842118 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2be8ea-f97f-49ac-8b08-6fc3d7c9aa3d None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:57.849685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 315/626] 10.222.0.16 () {56 vars in 1112 bytes} [Sun Oct 30 00:43:57 2022] GET /identity/v3/services?name=network => generated 124 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:57.850576 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c588513-f323-484a-9244-6696981412cd None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:57.879545 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a426c019-b4fd-4855-81df-99b05789a7db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:57.880568 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a426c019-b4fd-4855-81df-99b05789a7db None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:57.881033 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a426c019-b4fd-4855-81df-99b05789a7db None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:57.881509 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a426c019-b4fd-4855-81df-99b05789a7db None admin] PATH_INFO: `/v3/endpoints` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:57.882445 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a426c019-b4fd-4855-81df-99b05789a7db None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:57.884110 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a426c019-b4fd-4855-81df-99b05789a7db None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:57.893447 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:57.910777 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 312/627] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:43:57 2022] POST /identity/v3/endpoints => generated 346 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:43:57.912098 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a426c019-b4fd-4855-81df-99b05789a7db None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:57.922273 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:57.979141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c588513-f323-484a-9244-6696981412cd None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:58.043080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a426c019-b4fd-4855-81df-99b05789a7db None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:58.064688 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621c4db4-6c52-43b9-be92-617af540d2c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:58.065438 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621c4db4-6c52-43b9-be92-617af540d2c4 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:58.065719 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621c4db4-6c52-43b9-be92-617af540d2c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:58.065990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621c4db4-6c52-43b9-be92-617af540d2c4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:58.083492 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf3f51f-f227-4005-b7e0-dc91dc4383b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:58.084525 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf3f51f-f227-4005-b7e0-dc91dc4383b7 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:58.084992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf3f51f-f227-4005-b7e0-dc91dc4383b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:58.085467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf3f51f-f227-4005-b7e0-dc91dc4383b7 None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:58.086306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf3f51f-f227-4005-b7e0-dc91dc4383b7 None admin] RBAC: Adding query filter params (type=network) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:58.086791 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf3f51f-f227-4005-b7e0-dc91dc4383b7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:58.088410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf3f51f-f227-4005-b7e0-dc91dc4383b7 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:58.088605 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621c4db4-6c52-43b9-be92-617af540d2c4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:58.090032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621c4db4-6c52-43b9-be92-617af540d2c4 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:58.095121 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 313/628] 10.222.0.16 () {56 vars in 1112 bytes} [Sun Oct 30 00:43:57 2022] GET /identity/v3/services?type=network => generated 350 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:58.105751 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf3f51f-f227-4005-b7e0-dc91dc4383b7 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:58.115075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-621c4db4-6c52-43b9-be92-617af540d2c4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:58.115075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:58.115075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:58.115075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:58.115075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:58.115075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:58.115075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:58.115075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:58.115075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:58.115075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:58.115075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:58.115075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:58.115075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:43:58.115075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:43:58.115075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:43:58.115075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:43:58.116111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:58.116111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:58.116111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:58.116111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:58.116111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:58.116111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:58.116111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:58.116111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:58.116111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:58.116111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:58.116111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:58.116111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:58.116111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:58.116111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:58.116111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:58.116111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:58.116111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:43:58.117091 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:43:58.117091 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:43:58.117091 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:58.117272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 316/629] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:43:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:58.124532 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621c4db4-6c52-43b9-be92-617af540d2c4 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:58.182545 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:58.242639 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf3f51f-f227-4005-b7e0-dc91dc4383b7 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:58.245557 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621c4db4-6c52-43b9-be92-617af540d2c4 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:58.265104 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:58.287180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe4d3ce-37d8-4b85-944d-31aac6a402a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:58.287958 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe4d3ce-37d8-4b85-944d-31aac6a402a4 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:58.288303 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe4d3ce-37d8-4b85-944d-31aac6a402a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:58.288637 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe4d3ce-37d8-4b85-944d-31aac6a402a4 None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:58.289122 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d70012-09f2-46d5-96d3-2dd6209449d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:58.289432 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe4d3ce-37d8-4b85-944d-31aac6a402a4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:58.289842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe4d3ce-37d8-4b85-944d-31aac6a402a4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:58.290289 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d70012-09f2-46d5-96d3-2dd6209449d9 None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:58.290751 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d70012-09f2-46d5-96d3-2dd6209449d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:58.291187 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d70012-09f2-46d5-96d3-2dd6209449d9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:58.291519 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe4d3ce-37d8-4b85-944d-31aac6a402a4 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:58.292217 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d70012-09f2-46d5-96d3-2dd6209449d9 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:58.294552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d70012-09f2-46d5-96d3-2dd6209449d9 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:58.305362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 317/630] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:43:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:58.311358 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe4d3ce-37d8-4b85-944d-31aac6a402a4 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:58.321586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 314/631] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:43:58 2022] POST /identity/v3/endpoints => generated 337 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:43:58.458726 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe4d3ce-37d8-4b85-944d-31aac6a402a4 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:58.504951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87900346-d438-46d8-b3ce-8c0a0eb118e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:58.505604 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87900346-d438-46d8-b3ce-8c0a0eb118e3 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:58.505882 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87900346-d438-46d8-b3ce-8c0a0eb118e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:58.506214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87900346-d438-46d8-b3ce-8c0a0eb118e3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:58.513848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87900346-d438-46d8-b3ce-8c0a0eb118e3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:58.515330 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87900346-d438-46d8-b3ce-8c0a0eb118e3 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:58.531796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-87900346-d438-46d8-b3ce-8c0a0eb118e3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:58.531796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:58.531796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:58.531796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:58.531796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:58.531796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:58.531796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:58.531796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:58.531796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:58.531796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:58.531796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:58.531796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:58.531796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:43:58.531796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:43:58.531796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:43:58.532721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:43:58.532721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:58.532721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:58.532721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:58.532721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:58.532721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:58.532721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:58.532721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:58.532721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:58.532721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:58.532721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:58.532721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:58.532721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:58.532721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:58.532721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:58.532721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:58.532721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:58.533705 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:43:58.533705 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:43:58.533705 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:43:58.533705 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:43:58.533705 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:43:58.533705 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:43:58.533705 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:43:58.533705 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:58.538229 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 318/632] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:43:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:58.539502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d70012-09f2-46d5-96d3-2dd6209449d9 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:58.662393 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d70012-09f2-46d5-96d3-2dd6209449d9 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:58.673067 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:58.701217 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b29b8b8-ae25-4976-85be-61af6192e4b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:58.701836 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b29b8b8-ae25-4976-85be-61af6192e4b6 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:58.702102 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b29b8b8-ae25-4976-85be-61af6192e4b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:58.702364 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b29b8b8-ae25-4976-85be-61af6192e4b6 None admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:58.703055 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b29b8b8-ae25-4976-85be-61af6192e4b6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:58.703352 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b29b8b8-ae25-4976-85be-61af6192e4b6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:58.704797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b29b8b8-ae25-4976-85be-61af6192e4b6 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:58.721798 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 315/633] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:43:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:58.728065 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87900346-d438-46d8-b3ce-8c0a0eb118e3 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:58.779276 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:58.848329 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87900346-d438-46d8-b3ce-8c0a0eb118e3 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:58.884035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ad9a19-3866-4187-bc2f-5cdf39764c0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:58.884980 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ad9a19-3866-4187-bc2f-5cdf39764c0d None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:58.885443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ad9a19-3866-4187-bc2f-5cdf39764c0d None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:58.885898 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ad9a19-3866-4187-bc2f-5cdf39764c0d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:58.886813 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ad9a19-3866-4187-bc2f-5cdf39764c0d None admin] RBAC: Adding query filter params (role.id=5a0f6f642a10411da6fc1bfc92a7a3d3, scope.project.id=658799ed31d2472fb36af4aff0f03aab, user.id=4716e7afe2784e2a9b872996d895c374) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:43:58.887347 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ad9a19-3866-4187-bc2f-5cdf39764c0d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:58.888986 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ad9a19-3866-4187-bc2f-5cdf39764c0d None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:58.895446 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 319/634] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:43:58 2022] GET /identity/v3/role_assignments?user.id=4716e7afe2784e2a9b872996d895c374&scope.project.id=658799ed31d2472fb36af4aff0f03aab&role.id=5a0f6f642a10411da6fc1bfc92a7a3d3 => generated 608 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:43:59.209039 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a34f19-415e-487b-8f86-79e872b0ae87 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:59.209243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a34f19-415e-487b-8f86-79e872b0ae87 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:59.209538 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a34f19-415e-487b-8f86-79e872b0ae87 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:59.211239 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 316/635] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:59.217022 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5709ec9b-52a1-43b8-b128-c559f3584fda None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:59.217302 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5709ec9b-52a1-43b8-b128-c559f3584fda None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:59.217579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5709ec9b-52a1-43b8-b128-c559f3584fda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:59.236427 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-5709ec9b-52a1-43b8-b128-c559f3584fda None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:59.237401 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-5709ec9b-52a1-43b8-b128-c559f3584fda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:59.309277 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 320/636] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:59 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:43:59.389361 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb66f219-4f4d-4164-bd48-8d4117d6f121 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:59.389713 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb66f219-4f4d-4164-bd48-8d4117d6f121 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:59.389899 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb66f219-4f4d-4164-bd48-8d4117d6f121 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:59.391550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 317/637] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:43:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:43:59.397057 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cac806d-4b6b-4a25-bab5-34690dad82fa None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:59.397399 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cac806d-4b6b-4a25-bab5-34690dad82fa None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:59.397672 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cac806d-4b6b-4a25-bab5-34690dad82fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:59.416326 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-3cac806d-4b6b-4a25-bab5-34690dad82fa None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:43:59.417573 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3cac806d-4b6b-4a25-bab5-34690dad82fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:43:59.443422 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:43:59.487153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 321/638] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:43:59 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 30 00:43:59.590965 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb66f219-4f4d-4164-bd48-8d4117d6f121 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:59.693800 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb66f219-4f4d-4164-bd48-8d4117d6f121 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:59.727706 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5199bc-933c-4ad6-9621-66fb86fd82fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:59.728338 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5199bc-933c-4ad6-9621-66fb86fd82fd None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:59.728609 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5199bc-933c-4ad6-9621-66fb86fd82fd None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:59.728871 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5199bc-933c-4ad6-9621-66fb86fd82fd None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:59.729505 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5199bc-933c-4ad6-9621-66fb86fd82fd None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:59.730857 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5199bc-933c-4ad6-9621-66fb86fd82fd None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:59.747620 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 318/639] 10.222.0.16 () {60 vars in 1141 bytes} [Sun Oct 30 00:43:59 2022] POST /identity/v3/services => generated 269 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:43:59.756461 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cac806d-4b6b-4a25-bab5-34690dad82fa None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:59.873177 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cac806d-4b6b-4a25-bab5-34690dad82fa None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:43:59.917341 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e678f9-2a7d-4e63-8f56-c5f2580571e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:43:59.917971 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e678f9-2a7d-4e63-8f56-c5f2580571e5 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:43:59.918245 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e678f9-2a7d-4e63-8f56-c5f2580571e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:43:59.918511 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e678f9-2a7d-4e63-8f56-c5f2580571e5 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:43:59.919086 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e678f9-2a7d-4e63-8f56-c5f2580571e5 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:43:59.920471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e678f9-2a7d-4e63-8f56-c5f2580571e5 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:43:59.929776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-48e678f9-2a7d-4e63-8f56-c5f2580571e5 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 30 00:43:59.929776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:59.929776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:43:59.929776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:43:59.929776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:43:59.929776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:43:59.929776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:43:59.929776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:43:59.929776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 30 00:43:59.929776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:59.929776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:43:59.929776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:59.929776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:43:59.929776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:43:59.929776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:43:59.929776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:43:59.929776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:43:59.930772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:43:59.930772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:43:59.930772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:43:59.930772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:43:59.930772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:43:59.930772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:43:59.930772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:43:59.930772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:43:59.930772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:43:59.930772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:43:59.930772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:43:59.930772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:43:59.930772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:43:59.930772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:43:59.930772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:43:59.930772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:43:59.931816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:43:59.931816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:43:59.931816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:43:59.931816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:43:59.931816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:43:59.931816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:43:59.931816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:43:59.931816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:43:59.931816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:43:59.931816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:43:59.931816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:43:59.931816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:43:59.931816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 30 00:43:59.931816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:43:59.931816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 322/640] 10.222.0.16 () {56 vars in 1126 bytes} [Sun Oct 30 00:43:59 2022] GET /identity/v3/services/object-store => generated 93 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:43:59.938062 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5199bc-933c-4ad6-9621-66fb86fd82fd None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:43:59.969719 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:00.050916 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5199bc-933c-4ad6-9621-66fb86fd82fd None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:00.086288 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b5a4bb-a448-49fd-a6d6-4f1f48fc37df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:00.086923 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b5a4bb-a448-49fd-a6d6-4f1f48fc37df None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:00.087171 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b5a4bb-a448-49fd-a6d6-4f1f48fc37df None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:00.087438 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b5a4bb-a448-49fd-a6d6-4f1f48fc37df None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:00.088092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b5a4bb-a448-49fd-a6d6-4f1f48fc37df None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:00.088389 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b5a4bb-a448-49fd-a6d6-4f1f48fc37df None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:00.089749 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b5a4bb-a448-49fd-a6d6-4f1f48fc37df None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:00.094911 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 319/641] 10.222.0.16 () {56 vars in 1122 bytes} [Sun Oct 30 00:43:59 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:00.100685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e678f9-2a7d-4e63-8f56-c5f2580571e5 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:00.121720 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:00.209290 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e678f9-2a7d-4e63-8f56-c5f2580571e5 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:00.244949 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a9ba66-ffa9-4398-8822-2b5af8664d71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:00.246039 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a9ba66-ffa9-4398-8822-2b5af8664d71 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:00.246510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a9ba66-ffa9-4398-8822-2b5af8664d71 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:00.246973 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a9ba66-ffa9-4398-8822-2b5af8664d71 None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:00.247900 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a9ba66-ffa9-4398-8822-2b5af8664d71 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:00.248423 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a9ba66-ffa9-4398-8822-2b5af8664d71 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:00.249978 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a9ba66-ffa9-4398-8822-2b5af8664d71 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:00.255354 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 323/642] 10.222.0.16 () {56 vars in 1122 bytes} [Sun Oct 30 00:44:00 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:00.515411 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec0119f-2197-4d4c-b242-9cb198ec85fe None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:00.515816 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec0119f-2197-4d4c-b242-9cb198ec85fe None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:00.516014 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec0119f-2197-4d4c-b242-9cb198ec85fe None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:00.517563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 320/643] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:00.523020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04c02f4-118d-4c63-857a-82512c4d4447 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:00.523473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04c02f4-118d-4c63-857a-82512c4d4447 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:00.523549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04c02f4-118d-4c63-857a-82512c4d4447 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:00.542337 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-c04c02f4-118d-4c63-857a-82512c4d4447 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:00.543495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c04c02f4-118d-4c63-857a-82512c4d4447 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:00.616544 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 324/644] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:00 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 30 00:44:00.887342 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec0119f-2197-4d4c-b242-9cb198ec85fe None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:00.984362 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:00.997386 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec0119f-2197-4d4c-b242-9cb198ec85fe None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:01.032601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33b6af0-4954-493e-8e53-9064c76ba61e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:01.033446 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33b6af0-4954-493e-8e53-9064c76ba61e None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:01.033765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33b6af0-4954-493e-8e53-9064c76ba61e None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:01.034039 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33b6af0-4954-493e-8e53-9064c76ba61e None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:01.034708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33b6af0-4954-493e-8e53-9064c76ba61e None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:01.036325 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33b6af0-4954-493e-8e53-9064c76ba61e None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:01.046448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-f33b6af0-4954-493e-8e53-9064c76ba61e None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 30 00:44:01.046448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:01.046448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:44:01.046448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:44:01.046448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:44:01.046448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:44:01.046448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:44:01.046448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:01.046448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 30 00:44:01.046448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:01.046448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:44:01.046448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:01.046448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:01.046448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:01.046448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:01.046448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:01.046448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:01.047524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:01.047524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:01.047524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:01.047524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:01.047524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:01.047524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:01.047524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:44:01.047524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:44:01.047524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:44:01.047524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:44:01.047524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:01.047524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:01.047524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:01.047524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:01.047524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:01.047524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:01.048433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:01.048433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:01.048433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:01.048433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:01.048433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:01.048433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:01.048433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:01.048433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:01.048433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:01.048433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:01.048433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:44:01.048433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:01.048433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 30 00:44:01.048433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:01.048433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 321/645] 10.222.0.16 () {56 vars in 1114 bytes} [Sun Oct 30 00:44:00 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:01.054295 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04c02f4-118d-4c63-857a-82512c4d4447 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:01.070194 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:01.157883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04c02f4-118d-4c63-857a-82512c4d4447 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:01.163456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320462ee-98c4-46c8-9f71-cabbd07bfe25 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:01.164018 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320462ee-98c4-46c8-9f71-cabbd07bfe25 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:01.164505 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320462ee-98c4-46c8-9f71-cabbd07bfe25 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:01.166280 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 322/646] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:01 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:44:01.171294 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b145fe9-fa8e-4647-a6b7-4df3bb1973fa None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:01.171642 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b145fe9-fa8e-4647-a6b7-4df3bb1973fa None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:01.171943 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b145fe9-fa8e-4647-a6b7-4df3bb1973fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:01.190986 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-6b145fe9-fa8e-4647-a6b7-4df3bb1973fa None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:01.192311 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-6b145fe9-fa8e-4647-a6b7-4df3bb1973fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:01.197969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88586344-1c33-40a9-95fa-30cbbdade528 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:01.198826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88586344-1c33-40a9-95fa-30cbbdade528 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:01.199293 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88586344-1c33-40a9-95fa-30cbbdade528 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:01.199903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88586344-1c33-40a9-95fa-30cbbdade528 None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:01.200713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88586344-1c33-40a9-95fa-30cbbdade528 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:01.201198 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88586344-1c33-40a9-95fa-30cbbdade528 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:01.202856 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88586344-1c33-40a9-95fa-30cbbdade528 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:01.208321 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 325/647] 10.222.0.16 () {56 vars in 1114 bytes} [Sun Oct 30 00:44:01 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:01.213677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88586344-1c33-40a9-95fa-30cbbdade528 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:01.279797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 323/648] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:01 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:01.298982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:01.323557 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88586344-1c33-40a9-95fa-30cbbdade528 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:01.358609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7f747c-c7d3-47f0-a2ed-934480eb5f1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:01.359215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7f747c-c7d3-47f0-a2ed-934480eb5f1e None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:01.359490 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7f747c-c7d3-47f0-a2ed-934480eb5f1e None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:01.359779 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7f747c-c7d3-47f0-a2ed-934480eb5f1e None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:01.360403 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7f747c-c7d3-47f0-a2ed-934480eb5f1e None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:01.360698 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7f747c-c7d3-47f0-a2ed-934480eb5f1e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:01.362209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7f747c-c7d3-47f0-a2ed-934480eb5f1e None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:01.367075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 326/649] 10.222.0.16 () {56 vars in 1114 bytes} [Sun Oct 30 00:44:01 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:01.554603 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b145fe9-fa8e-4647-a6b7-4df3bb1973fa None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:01.644742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:01.657588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b145fe9-fa8e-4647-a6b7-4df3bb1973fa None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:01.692697 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f64ce7b-2de5-4923-a2fa-afc148cd75c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:01.693706 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f64ce7b-2de5-4923-a2fa-afc148cd75c5 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:01.694245 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f64ce7b-2de5-4923-a2fa-afc148cd75c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:01.694716 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f64ce7b-2de5-4923-a2fa-afc148cd75c5 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:01.705630 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f64ce7b-2de5-4923-a2fa-afc148cd75c5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:44:01.710745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-5f64ce7b-2de5-4923-a2fa-afc148cd75c5 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 30 00:44:01.710745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:01.710745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:01.710745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:01.710745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:01.710745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:01.710745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:01.710745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:01.710745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:01.710745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:01.710745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:01.710745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:01.710745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:44:01.710745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:44:01.710745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:44:01.710745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:44:01.711643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:44:01.711643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:44:01.711643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:44:01.711643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:44:01.711643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:01.711643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:01.711643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:01.711643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:01.711643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:01.711643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:01.711643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:01.711643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:01.711643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:01.711643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:01.711643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:01.711643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:01.712515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:01.712515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:01.712515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:01.712515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:01.712515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:44:01.712515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:44:01.712515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:44:01.712515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:44:01.712515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:44:01.712515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:44:01.712515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 30 00:44:01.712515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:01.719698 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 324/650] 10.222.0.16 () {56 vars in 1120 bytes} [Sun Oct 30 00:44:01 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:01.721370 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7f747c-c7d3-47f0-a2ed-934480eb5f1e None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:01.822469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7f747c-c7d3-47f0-a2ed-934480eb5f1e None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:01.858748 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf827dd-85fd-46d3-bb8e-d60317bed573 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:01.859357 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf827dd-85fd-46d3-bb8e-d60317bed573 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:01.859632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf827dd-85fd-46d3-bb8e-d60317bed573 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:01.859940 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf827dd-85fd-46d3-bb8e-d60317bed573 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:01.860593 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf827dd-85fd-46d3-bb8e-d60317bed573 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:01.860892 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf827dd-85fd-46d3-bb8e-d60317bed573 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:01.862409 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf827dd-85fd-46d3-bb8e-d60317bed573 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:01.871198 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 327/651] 10.222.0.16 () {56 vars in 1115 bytes} [Sun Oct 30 00:44:01 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:01.876930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f64ce7b-2de5-4923-a2fa-afc148cd75c5 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:01.983506 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f64ce7b-2de5-4923-a2fa-afc148cd75c5 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:02.009549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc90823-50ca-4507-b97a-0134a61523cb None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:02.009858 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc90823-50ca-4507-b97a-0134a61523cb None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:02.010089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc90823-50ca-4507-b97a-0134a61523cb None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:02.011684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 328/652] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:02.016678 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42eaa946-791c-44b1-9a48-c2105af32cb8 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:02.016958 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42eaa946-791c-44b1-9a48-c2105af32cb8 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:02.017232 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42eaa946-791c-44b1-9a48-c2105af32cb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:02.021400 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e81d856-0740-4193-96ae-67230af61a51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:02.022315 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e81d856-0740-4193-96ae-67230af61a51 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:02.022782 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e81d856-0740-4193-96ae-67230af61a51 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:02.023276 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e81d856-0740-4193-96ae-67230af61a51 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:02.032841 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:02.034197 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e81d856-0740-4193-96ae-67230af61a51 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:02.035703 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e81d856-0740-4193-96ae-67230af61a51 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:02.036425 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-42eaa946-791c-44b1-9a48-c2105af32cb8 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:02.038850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-42eaa946-791c-44b1-9a48-c2105af32cb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:02.044438 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:02.052544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-8e81d856-0740-4193-96ae-67230af61a51 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:44:02.052544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:02.052544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:02.052544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:02.052544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:02.052544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:02.052544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:02.052544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:02.052544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:02.052544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:02.052544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:02.052544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:02.052544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:44:02.052544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:44:02.052544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:44:02.052544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:44:02.053694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:02.053694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:02.053694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:02.053694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:02.053694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:02.053694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:02.053694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:02.053694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:02.053694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:02.053694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:02.053694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:02.053694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:02.053694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:02.053694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:02.053694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:02.053694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:02.053694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:44:02.054827 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:44:02.054827 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:44:02.054827 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:02.056686 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 325/653] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:44:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:02.062670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e81d856-0740-4193-96ae-67230af61a51 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:02.124137 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 329/654] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:02 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 30 00:44:02.167680 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e81d856-0740-4193-96ae-67230af61a51 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:02.202376 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ebe0a6-5677-404c-90b3-dffc286f8e5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:02.202979 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ebe0a6-5677-404c-90b3-dffc286f8e5f None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:02.203253 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ebe0a6-5677-404c-90b3-dffc286f8e5f None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:02.203517 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ebe0a6-5677-404c-90b3-dffc286f8e5f None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:02.204185 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ebe0a6-5677-404c-90b3-dffc286f8e5f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:02.204480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ebe0a6-5677-404c-90b3-dffc286f8e5f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:02.206029 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ebe0a6-5677-404c-90b3-dffc286f8e5f None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:02.223409 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 326/655] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:44:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:44:02.224801 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42eaa946-791c-44b1-9a48-c2105af32cb8 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:02.323086 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42eaa946-791c-44b1-9a48-c2105af32cb8 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:02.355197 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:02.360381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e983517-08d0-43a0-99b8-06216a2fce7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:02.361026 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e983517-08d0-43a0-99b8-06216a2fce7a None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:02.361335 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e983517-08d0-43a0-99b8-06216a2fce7a None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:02.361604 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e983517-08d0-43a0-99b8-06216a2fce7a None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:02.371790 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e983517-08d0-43a0-99b8-06216a2fce7a None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:02.371790 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e983517-08d0-43a0-99b8-06216a2fce7a None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:02.382851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-4e983517-08d0-43a0-99b8-06216a2fce7a None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 30 00:44:02.382851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:02.382851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:02.382851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:02.382851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:02.382851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:02.382851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:02.382851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:02.382851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:02.382851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:02.382851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:02.382851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:02.382851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:44:02.382851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:44:02.382851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:44:02.382851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:44:02.383969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:02.383969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:02.383969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:44:02.383969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:44:02.383969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:44:02.383969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:44:02.383969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:02.383969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:02.383969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:02.383969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:02.383969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:02.383969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:02.383969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:02.383969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:02.383969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:02.383969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:02.383969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:02.383969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:02.385131 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:02.385131 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:02.385131 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:44:02.385131 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:44:02.385131 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:44:02.385131 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:44:02.385131 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:44:02.385131 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:44:02.385131 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 30 00:44:02.385131 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:02.387211 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 330/656] 10.222.0.16 () {56 vars in 1117 bytes} [Sun Oct 30 00:44:02 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:02.392925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ebe0a6-5677-404c-90b3-dffc286f8e5f None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:02.423885 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e983517-08d0-43a0-99b8-06216a2fce7a None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:02.455634 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:02.499396 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ebe0a6-5677-404c-90b3-dffc286f8e5f None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:02.539450 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d239d8-3fe9-454f-92f7-dc56c92e8a1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:02.540191 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d239d8-3fe9-454f-92f7-dc56c92e8a1e None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:02.540457 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d239d8-3fe9-454f-92f7-dc56c92e8a1e None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:02.540717 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d239d8-3fe9-454f-92f7-dc56c92e8a1e None admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:02.541440 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d239d8-3fe9-454f-92f7-dc56c92e8a1e None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:02.541734 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d239d8-3fe9-454f-92f7-dc56c92e8a1e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:02.543358 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d239d8-3fe9-454f-92f7-dc56c92e8a1e None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:02.546216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e983517-08d0-43a0-99b8-06216a2fce7a None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:02.562869 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 327/657] 10.222.0.16 () {56 vars in 1149 bytes} [Sun Oct 30 00:44:02 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:02.569678 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d239d8-3fe9-454f-92f7-dc56c92e8a1e None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:02.585274 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b6fbc3-4d03-4658-a87d-ee6c2e6034e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:02.585991 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b6fbc3-4d03-4658-a87d-ee6c2e6034e6 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:02.586351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b6fbc3-4d03-4658-a87d-ee6c2e6034e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:02.586681 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b6fbc3-4d03-4658-a87d-ee6c2e6034e6 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:02.587321 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b6fbc3-4d03-4658-a87d-ee6c2e6034e6 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:02.588796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b6fbc3-4d03-4658-a87d-ee6c2e6034e6 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:02.597755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-66b6fbc3-4d03-4658-a87d-ee6c2e6034e6 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 30 00:44:02.597755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:02.597755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:44:02.597755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:44:02.597755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:44:02.597755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:44:02.597755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:44:02.597755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:02.597755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 30 00:44:02.597755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:02.597755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:44:02.597755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:02.597755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:02.597755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:02.597755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:02.597755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:02.598916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:02.598916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:02.598916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:02.598916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:02.598916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:02.598916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:02.598916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:02.598916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:44:02.598916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:44:02.598916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:44:02.598916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:44:02.598916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:02.598916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:02.598916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:02.598916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:02.598916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:02.598916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:02.599994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:02.599994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:02.599994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:02.599994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:02.599994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:02.599994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:02.599994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:02.599994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:02.599994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:02.599994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:02.599994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:44:02.599994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:02.599994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 30 00:44:02.599994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:02.600718 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 331/658] 10.222.0.16 () {56 vars in 1132 bytes} [Sun Oct 30 00:44:02 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:44:02.602692 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25926729-e93f-45e5-8072-ba11c57459c9 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:02.603034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25926729-e93f-45e5-8072-ba11c57459c9 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:02.603367 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25926729-e93f-45e5-8072-ba11c57459c9 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:02.605048 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 332/659] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:44:02.606502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25926729-e93f-45e5-8072-ba11c57459c9 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:02.667201 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:02.681743 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d239d8-3fe9-454f-92f7-dc56c92e8a1e None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:02.699962 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:02.717850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8832771f-b28c-40b5-8d7f-fa0604ed32c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:02.718321 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25926729-e93f-45e5-8072-ba11c57459c9 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:02.718494 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8832771f-b28c-40b5-8d7f-fa0604ed32c8 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:02.718738 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8832771f-b28c-40b5-8d7f-fa0604ed32c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:02.719000 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8832771f-b28c-40b5-8d7f-fa0604ed32c8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:02.731489 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8832771f-b28c-40b5-8d7f-fa0604ed32c8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:02.733288 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8832771f-b28c-40b5-8d7f-fa0604ed32c8 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:02.748651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-8832771f-b28c-40b5-8d7f-fa0604ed32c8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:44:02.748651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:02.748651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:02.748651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:02.748651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:02.748651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:02.748651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:02.748651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:02.748651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:02.748651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:02.748651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:02.748651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:02.748651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:44:02.748651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:44:02.748651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:44:02.748651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:44:02.749627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:02.749627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:02.749627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:02.749627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:02.749627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:02.749627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:02.749627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:02.749627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:02.749627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:02.749627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:02.749627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:02.749627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:02.749627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:02.749627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:02.749627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:02.749627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:02.749627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:44:02.750518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:44:02.750518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:44:02.750518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:02.750518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 328/660] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:44:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:44:02.751538 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c087c8-cc8f-483c-ad95-bb467e246536 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:02.751538 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c087c8-cc8f-483c-ad95-bb467e246536 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:02.751538 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c087c8-cc8f-483c-ad95-bb467e246536 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:02.759850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda78849-217c-4a90-a602-59279c6d1712 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:02.760464 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda78849-217c-4a90-a602-59279c6d1712 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:02.760733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda78849-217c-4a90-a602-59279c6d1712 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:02.760995 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda78849-217c-4a90-a602-59279c6d1712 None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:02.761621 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda78849-217c-4a90-a602-59279c6d1712 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:02.761909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda78849-217c-4a90-a602-59279c6d1712 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:02.763264 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda78849-217c-4a90-a602-59279c6d1712 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:02.773312 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 333/661] 10.222.0.16 () {56 vars in 1126 bytes} [Sun Oct 30 00:44:02 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:44:02.773477 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-f5c087c8-cc8f-483c-ad95-bb467e246536 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:02.773562 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda78849-217c-4a90-a602-59279c6d1712 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:02.773634 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-f5c087c8-cc8f-483c-ad95-bb467e246536 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:02.839812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 329/662] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:02 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:02.839812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c087c8-cc8f-483c-ad95-bb467e246536 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:02.887990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda78849-217c-4a90-a602-59279c6d1712 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:02.915164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:02.933531 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d228b4f-a20e-41ef-b820-1e87189198b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:02.934063 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d228b4f-a20e-41ef-b820-1e87189198b0 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:02.934335 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d228b4f-a20e-41ef-b820-1e87189198b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:02.934598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d228b4f-a20e-41ef-b820-1e87189198b0 None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:02.935647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d228b4f-a20e-41ef-b820-1e87189198b0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:02.936440 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d228b4f-a20e-41ef-b820-1e87189198b0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:02.938112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d228b4f-a20e-41ef-b820-1e87189198b0 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:02.956868 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 334/663] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:44:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:02.958328 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d228b4f-a20e-41ef-b820-1e87189198b0 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:02.959923 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:02.969215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c087c8-cc8f-483c-ad95-bb467e246536 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:03.005908 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428e95f7-8f84-4383-831a-ce5b406b29f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:03.007305 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428e95f7-8f84-4383-831a-ce5b406b29f7 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:03.007940 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428e95f7-8f84-4383-831a-ce5b406b29f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:03.008397 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428e95f7-8f84-4383-831a-ce5b406b29f7 None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:03.009303 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428e95f7-8f84-4383-831a-ce5b406b29f7 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:03.009836 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428e95f7-8f84-4383-831a-ce5b406b29f7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:03.011511 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428e95f7-8f84-4383-831a-ce5b406b29f7 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:03.018067 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 330/664] 10.222.0.16 () {56 vars in 1126 bytes} [Sun Oct 30 00:44:02 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:03.026143 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428e95f7-8f84-4383-831a-ce5b406b29f7 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:03.065835 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d228b4f-a20e-41ef-b820-1e87189198b0 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:03.106773 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e60cc1e-e519-4cc8-8074-a5cf2db7b24b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:03.107494 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e60cc1e-e519-4cc8-8074-a5cf2db7b24b None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:03.107803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e60cc1e-e519-4cc8-8074-a5cf2db7b24b None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:03.108072 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e60cc1e-e519-4cc8-8074-a5cf2db7b24b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:03.115506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e60cc1e-e519-4cc8-8074-a5cf2db7b24b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:03.116962 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e60cc1e-e519-4cc8-8074-a5cf2db7b24b None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:03.128209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-4e60cc1e-e519-4cc8-8074-a5cf2db7b24b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:44:03.128209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:03.128209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:03.128209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:03.128209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:03.128209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:03.128209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:03.128209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:03.128209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:03.128209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:03.128209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:03.128209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:03.128209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:44:03.128209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:44:03.128209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:44:03.129300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:44:03.129300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:03.129300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:03.129300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:03.129300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:03.129300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:03.129300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:03.129300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:03.129300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:03.129300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:03.129300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:03.129300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:03.129300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:03.129300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:03.129300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:03.129300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:03.129300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:03.132903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:44:03.132903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:44:03.132903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:44:03.132903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:44:03.132903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:44:03.132903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:44:03.132903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:44:03.132903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:03.132903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 335/665] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:44:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:03.132903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e60cc1e-e519-4cc8-8074-a5cf2db7b24b None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:03.157427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428e95f7-8f84-4383-831a-ce5b406b29f7 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:03.175010 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:03.198216 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22261c09-3ea3-4da5-99a3-52923319c319 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:03.198743 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22261c09-3ea3-4da5-99a3-52923319c319 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:03.199016 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22261c09-3ea3-4da5-99a3-52923319c319 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:03.199334 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22261c09-3ea3-4da5-99a3-52923319c319 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:03.200355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22261c09-3ea3-4da5-99a3-52923319c319 None admin] RBAC: Adding query filter params (interface=public, service_id=8dc82f660288453497488a57c557bb87, region_id=RegionOne) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:03.201219 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22261c09-3ea3-4da5-99a3-52923319c319 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:03.202858 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22261c09-3ea3-4da5-99a3-52923319c319 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:03.215866 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 331/666] 10.222.0.16 () {56 vars in 1251 bytes} [Sun Oct 30 00:44:03 2022] GET /identity/v3/endpoints?service_id=8dc82f660288453497488a57c557bb87&interface=public®ion_id=RegionOne => generated 194 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:03.216830 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22261c09-3ea3-4da5-99a3-52923319c319 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:03.241070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e60cc1e-e519-4cc8-8074-a5cf2db7b24b None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:03.246397 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:03.281569 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c753478c-20e9-4742-9134-d864ef7eb9f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:03.282178 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c753478c-20e9-4742-9134-d864ef7eb9f8 None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:03.282451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c753478c-20e9-4742-9134-d864ef7eb9f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:03.282713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c753478c-20e9-4742-9134-d864ef7eb9f8 None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:03.283822 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c753478c-20e9-4742-9134-d864ef7eb9f8 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:03.285203 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c753478c-20e9-4742-9134-d864ef7eb9f8 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:03.301139 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 336/667] 10.222.0.16 () {60 vars in 1141 bytes} [Sun Oct 30 00:44:03 2022] POST /identity/v3/services => generated 241 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:44:03.301783 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c753478c-20e9-4742-9134-d864ef7eb9f8 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:03.330318 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22261c09-3ea3-4da5-99a3-52923319c319 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:03.384940 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51dcb706-7bfc-4d0d-b073-9c380fbbb012 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:03.385958 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51dcb706-7bfc-4d0d-b073-9c380fbbb012 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:03.386424 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51dcb706-7bfc-4d0d-b073-9c380fbbb012 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:03.386877 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51dcb706-7bfc-4d0d-b073-9c380fbbb012 None admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:03.387813 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51dcb706-7bfc-4d0d-b073-9c380fbbb012 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:03.388324 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51dcb706-7bfc-4d0d-b073-9c380fbbb012 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:03.389993 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51dcb706-7bfc-4d0d-b073-9c380fbbb012 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:03.409432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 332/668] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:44:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:03.410695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51dcb706-7bfc-4d0d-b073-9c380fbbb012 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:03.436682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:03.440597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c753478c-20e9-4742-9134-d864ef7eb9f8 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:03.488142 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3ffebe-1f6b-4399-b63a-64af75407b49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:03.488674 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3ffebe-1f6b-4399-b63a-64af75407b49 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:03.488950 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3ffebe-1f6b-4399-b63a-64af75407b49 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:03.489215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3ffebe-1f6b-4399-b63a-64af75407b49 None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:03.489818 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3ffebe-1f6b-4399-b63a-64af75407b49 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:03.491898 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3ffebe-1f6b-4399-b63a-64af75407b49 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:03.497688 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 337/669] 10.222.0.16 () {56 vars in 1087 bytes} [Sun Oct 30 00:44:03 2022] GET /identity/v3/services => generated 1258 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:44:03.506851 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:03.530751 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51dcb706-7bfc-4d0d-b073-9c380fbbb012 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:03.579823 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4405b3e9-bf8d-4af7-9f91-4c7033bc91f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:03.580997 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4405b3e9-bf8d-4af7-9f91-4c7033bc91f9 None admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:03.581515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4405b3e9-bf8d-4af7-9f91-4c7033bc91f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:03.582024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4405b3e9-bf8d-4af7-9f91-4c7033bc91f9 None admin] PATH_INFO: `/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/4716e7afe2784e2a9b872996d895c374/roles/2aa395773c0a4706897ec0992f3a769a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:03.585017 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4405b3e9-bf8d-4af7-9f91-4c7033bc91f9 None admin] RBAC: Authorizing `identity:create_grant(project_id=658799ed31d2472fb36af4aff0f03aab, user_id=4716e7afe2784e2a9b872996d895c374, role_id=2aa395773c0a4706897ec0992f3a769a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:03.586792 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4405b3e9-bf8d-4af7-9f91-4c7033bc91f9 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:03.604470 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 333/670] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:44:03 2022] PUT /identity/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/4716e7afe2784e2a9b872996d895c374/roles/2aa395773c0a4706897ec0992f3a769a => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:44:03.656661 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0683326-2428-416c-8b25-fce827ff8989 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:03.657319 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0683326-2428-416c-8b25-fce827ff8989 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:03.657764 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0683326-2428-416c-8b25-fce827ff8989 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:03.659981 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 338/671] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:03 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:03.666042 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5d681d-85d5-4e3b-b48f-a595ad177356 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:03.666425 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5d681d-85d5-4e3b-b48f-a595ad177356 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:03.666655 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5d681d-85d5-4e3b-b48f-a595ad177356 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:03.687300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-8d5d681d-85d5-4e3b-b48f-a595ad177356 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:03.688696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-8d5d681d-85d5-4e3b-b48f-a595ad177356 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:03.766768 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 334/672] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:03 2022] POST /identity/v3/auth/tokens => generated 1373 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:04.030686 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0683326-2428-416c-8b25-fce827ff8989 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:04.136033 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0683326-2428-416c-8b25-fce827ff8989 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:04.171833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9667c48a-0cd6-452c-9fab-c69cec754bab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:04.172438 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9667c48a-0cd6-452c-9fab-c69cec754bab None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:04.172709 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9667c48a-0cd6-452c-9fab-c69cec754bab None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:04.172980 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9667c48a-0cd6-452c-9fab-c69cec754bab None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:04.173685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9667c48a-0cd6-452c-9fab-c69cec754bab None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:04.175060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9667c48a-0cd6-452c-9fab-c69cec754bab None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:04.191894 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 339/673] 10.222.0.16 () {60 vars in 1141 bytes} [Sun Oct 30 00:44:04 2022] POST /identity/v3/services => generated 251 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:44:05.638744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650b1a78-5839-4dbe-b471-c8451443fa29 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:05.639303 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650b1a78-5839-4dbe-b471-c8451443fa29 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:05.639303 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650b1a78-5839-4dbe-b471-c8451443fa29 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:05.640940 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 335/674] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:05.647342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1407f7-3961-49b5-8aa6-c0f60ac627f3 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:05.647943 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1407f7-3961-49b5-8aa6-c0f60ac627f3 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:05.647943 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1407f7-3961-49b5-8aa6-c0f60ac627f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:05.667521 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-af1407f7-3961-49b5-8aa6-c0f60ac627f3 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:05.668781 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-af1407f7-3961-49b5-8aa6-c0f60ac627f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:05.687955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:05.696004 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc7fabf-8ec9-463b-af1b-d75012abc99a None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:05.696281 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc7fabf-8ec9-463b-af1b-d75012abc99a None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:05.696557 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc7fabf-8ec9-463b-af1b-d75012abc99a None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:05.697897 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 336/675] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:05.703014 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d263184b-2bdb-4c4e-8eb7-1bddfe48bc84 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:05.703292 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d263184b-2bdb-4c4e-8eb7-1bddfe48bc84 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:05.703583 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d263184b-2bdb-4c4e-8eb7-1bddfe48bc84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:05.721500 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-d263184b-2bdb-4c4e-8eb7-1bddfe48bc84 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:05.722667 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d263184b-2bdb-4c4e-8eb7-1bddfe48bc84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:05.752970 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 340/676] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:05 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:05.769059 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fed5b3-e7b9-4e0e-acf9-bd14262b9b51 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:05.769376 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fed5b3-e7b9-4e0e-acf9-bd14262b9b51 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:05.769611 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fed5b3-e7b9-4e0e-acf9-bd14262b9b51 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:05.772289 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 341/677] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:44:05.776452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247cabb8-0437-4852-90af-8ef76ea98550 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:05.776831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247cabb8-0437-4852-90af-8ef76ea98550 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:05.777100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247cabb8-0437-4852-90af-8ef76ea98550 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:05.796124 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-247cabb8-0437-4852-90af-8ef76ea98550 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:05.797294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-247cabb8-0437-4852-90af-8ef76ea98550 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:05.800484 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 337/678] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:05 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:05.868446 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 342/679] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:05 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 30 00:44:06.045917 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d263184b-2bdb-4c4e-8eb7-1bddfe48bc84 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:06.049861 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:06.064925 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247cabb8-0437-4852-90af-8ef76ea98550 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:06.143394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:06.162710 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247cabb8-0437-4852-90af-8ef76ea98550 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:06.171449 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d263184b-2bdb-4c4e-8eb7-1bddfe48bc84 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:06.196447 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43050050-3bdd-4c52-9939-f85891e9d135 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:06.197099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43050050-3bdd-4c52-9939-f85891e9d135 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:06.197372 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43050050-3bdd-4c52-9939-f85891e9d135 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:06.197626 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43050050-3bdd-4c52-9939-f85891e9d135 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:06.198194 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43050050-3bdd-4c52-9939-f85891e9d135 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:06.199520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43050050-3bdd-4c52-9939-f85891e9d135 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:06.208241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-43050050-3bdd-4c52-9939-f85891e9d135 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 30 00:44:06.208241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:06.208241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:44:06.208241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:44:06.208241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:44:06.208241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:44:06.208241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:44:06.208241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:06.208241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 30 00:44:06.208241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:06.208241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:44:06.208241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:06.208241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:06.208241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:06.208241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:06.208241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:06.209215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:06.209215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:06.209215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:06.209215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:06.209215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:06.209215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:06.209215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:06.209215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:44:06.209215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:44:06.209215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:44:06.209215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:44:06.209215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:06.209215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:06.209215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:06.209215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:06.209215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:06.209215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:06.210113 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:06.210113 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:06.210113 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:06.210113 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:06.210113 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:06.210113 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:06.210113 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:06.210113 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:06.210113 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:06.210113 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:06.210113 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:44:06.210113 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:06.210113 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 30 00:44:06.210113 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:06.210113 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 343/680] 10.222.0.16 () {56 vars in 1132 bytes} [Sun Oct 30 00:44:06 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 145 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:06.210969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43050050-3bdd-4c52-9939-f85891e9d135 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:06.218659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52db30bc-ee1a-4a3e-a1e4-fe980e64c626 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:06.219517 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52db30bc-ee1a-4a3e-a1e4-fe980e64c626 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:06.219987 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52db30bc-ee1a-4a3e-a1e4-fe980e64c626 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:06.220589 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52db30bc-ee1a-4a3e-a1e4-fe980e64c626 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:06.221513 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52db30bc-ee1a-4a3e-a1e4-fe980e64c626 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:06.223504 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52db30bc-ee1a-4a3e-a1e4-fe980e64c626 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:06.234977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-52db30bc-ee1a-4a3e-a1e4-fe980e64c626 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 30 00:44:06.234977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:06.234977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:44:06.234977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:44:06.234977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:44:06.234977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:44:06.234977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:44:06.234977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:06.234977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 30 00:44:06.234977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:06.234977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:44:06.234977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:06.234977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:06.234977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:06.234977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:06.234977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:06.234977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:06.236300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:06.236300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:06.236300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:06.236300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:06.236300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:06.236300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:06.236300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:44:06.236300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:44:06.236300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:44:06.236300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:44:06.236300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:06.236300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:06.236300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:06.236300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:06.236300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:06.236300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:06.237232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:06.237232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:06.237232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:06.237232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:06.237232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:06.237232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:06.237232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:06.237232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:06.237232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:06.237232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:06.237232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:44:06.237232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:06.237232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 30 00:44:06.237232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:06.237952 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 338/681] 10.222.0.16 () {56 vars in 1126 bytes} [Sun Oct 30 00:44:06 2022] GET /identity/v3/services/object-store => generated 93 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:06.239000 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52db30bc-ee1a-4a3e-a1e4-fe980e64c626 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:06.320046 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:06.323942 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43050050-3bdd-4c52-9939-f85891e9d135 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:06.359372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52db30bc-ee1a-4a3e-a1e4-fe980e64c626 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:06.365004 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3dd1eb-ff7c-40ef-b630-32d9a343333f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:06.365659 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3dd1eb-ff7c-40ef-b630-32d9a343333f None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:06.365931 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3dd1eb-ff7c-40ef-b630-32d9a343333f None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:06.366256 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3dd1eb-ff7c-40ef-b630-32d9a343333f None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:06.378014 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3dd1eb-ff7c-40ef-b630-32d9a343333f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:44:06.383369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-db3dd1eb-ff7c-40ef-b630-32d9a343333f None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 30 00:44:06.383369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:06.383369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:06.383369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:06.383369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:06.383369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:06.383369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:06.383369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:06.383369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:06.383369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:06.383369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:06.383369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:06.383369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:44:06.383369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:44:06.383369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:44:06.383369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:44:06.384483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:44:06.384483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:44:06.384483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:44:06.384483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:44:06.384483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:06.384483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:06.384483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:06.384483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:06.384483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:06.384483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:06.384483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:06.384483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:06.384483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:06.384483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:06.384483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:06.384483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:06.385502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:06.385502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:06.385502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:06.385502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:06.385502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:44:06.385502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:44:06.385502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:44:06.385502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:44:06.385502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:44:06.385502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:44:06.385502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 30 00:44:06.385502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:06.385502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 344/682] 10.222.0.16 () {56 vars in 1120 bytes} [Sun Oct 30 00:44:06 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:06.385502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3dd1eb-ff7c-40ef-b630-32d9a343333f None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:06.389657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:06.398161 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52c8bdf-d402-42af-b16d-c5a785eeca1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:06.398764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52c8bdf-d402-42af-b16d-c5a785eeca1b None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:06.399039 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52c8bdf-d402-42af-b16d-c5a785eeca1b None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:06.399303 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52c8bdf-d402-42af-b16d-c5a785eeca1b None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:06.399948 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52c8bdf-d402-42af-b16d-c5a785eeca1b None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:06.400242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52c8bdf-d402-42af-b16d-c5a785eeca1b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:06.401591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52c8bdf-d402-42af-b16d-c5a785eeca1b None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:06.406224 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 339/683] 10.222.0.16 () {56 vars in 1126 bytes} [Sun Oct 30 00:44:06 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:06.407323 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1103f8-b6b0-4c5f-a9af-c5293d99596e None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:06.407594 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1103f8-b6b0-4c5f-a9af-c5293d99596e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:06.407894 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1103f8-b6b0-4c5f-a9af-c5293d99596e None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:06.409472 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 340/684] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:06.410150 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1103f8-b6b0-4c5f-a9af-c5293d99596e None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:06.487622 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3dd1eb-ff7c-40ef-b630-32d9a343333f None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:06.508540 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1103f8-b6b0-4c5f-a9af-c5293d99596e None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:06.521181 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36a7d57-4387-4e04-8b35-a675e73c08c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:06.521799 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36a7d57-4387-4e04-8b35-a675e73c08c8 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:06.522079 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36a7d57-4387-4e04-8b35-a675e73c08c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:06.522344 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36a7d57-4387-4e04-8b35-a675e73c08c8 None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:06.522979 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36a7d57-4387-4e04-8b35-a675e73c08c8 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:06.523272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36a7d57-4387-4e04-8b35-a675e73c08c8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:06.524669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36a7d57-4387-4e04-8b35-a675e73c08c8 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:06.529442 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 345/685] 10.222.0.16 () {56 vars in 1122 bytes} [Sun Oct 30 00:44:06 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:06.530355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36a7d57-4387-4e04-8b35-a675e73c08c8 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:06.537471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:06.543286 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92df09fb-5f9b-41b5-8bd5-34df03658218 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:06.543925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92df09fb-5f9b-41b5-8bd5-34df03658218 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:06.544185 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92df09fb-5f9b-41b5-8bd5-34df03658218 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:06.544457 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92df09fb-5f9b-41b5-8bd5-34df03658218 None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:06.545119 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92df09fb-5f9b-41b5-8bd5-34df03658218 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:06.545413 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92df09fb-5f9b-41b5-8bd5-34df03658218 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:06.546769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92df09fb-5f9b-41b5-8bd5-34df03658218 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:06.555562 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 341/686] 10.222.0.16 () {56 vars in 1115 bytes} [Sun Oct 30 00:44:06 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:06.556770 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9797a2c-cf1e-45cd-8571-86454a59fe51 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:06.557058 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9797a2c-cf1e-45cd-8571-86454a59fe51 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:06.557313 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9797a2c-cf1e-45cd-8571-86454a59fe51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:06.576025 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-a9797a2c-cf1e-45cd-8571-86454a59fe51 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:06.577124 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a9797a2c-cf1e-45cd-8571-86454a59fe51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:06.593933 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:06.627362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36a7d57-4387-4e04-8b35-a675e73c08c8 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:06.642016 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 342/687] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:06 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:06.642890 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9797a2c-cf1e-45cd-8571-86454a59fe51 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:06.674963 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456608f4-02ef-49dc-af67-8e700dd142c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:06.675668 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456608f4-02ef-49dc-af67-8e700dd142c4 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:06.676036 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456608f4-02ef-49dc-af67-8e700dd142c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:06.676306 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456608f4-02ef-49dc-af67-8e700dd142c4 None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:06.676922 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456608f4-02ef-49dc-af67-8e700dd142c4 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:06.677214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456608f4-02ef-49dc-af67-8e700dd142c4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:06.678572 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456608f4-02ef-49dc-af67-8e700dd142c4 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:06.683656 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 346/688] 10.222.0.16 () {56 vars in 1126 bytes} [Sun Oct 30 00:44:06 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:06.684332 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456608f4-02ef-49dc-af67-8e700dd142c4 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:06.748474 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9797a2c-cf1e-45cd-8571-86454a59fe51 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:06.771833 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:06.783649 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520fbd8c-c427-409a-ac36-c7fd34968223 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:06.784469 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520fbd8c-c427-409a-ac36-c7fd34968223 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:06.784926 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520fbd8c-c427-409a-ac36-c7fd34968223 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:06.785197 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520fbd8c-c427-409a-ac36-c7fd34968223 None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:06.785826 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520fbd8c-c427-409a-ac36-c7fd34968223 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:06.786120 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520fbd8c-c427-409a-ac36-c7fd34968223 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:06.787512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520fbd8c-c427-409a-ac36-c7fd34968223 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:06.787792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456608f4-02ef-49dc-af67-8e700dd142c4 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:06.792392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 343/689] 10.222.0.16 () {56 vars in 1122 bytes} [Sun Oct 30 00:44:06 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:06.793137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520fbd8c-c427-409a-ac36-c7fd34968223 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:06.821635 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:06.830011 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38eb7f80-5ee1-4bd6-96fa-cc82f650204e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:06.830636 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38eb7f80-5ee1-4bd6-96fa-cc82f650204e None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:06.830910 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38eb7f80-5ee1-4bd6-96fa-cc82f650204e None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:06.831175 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38eb7f80-5ee1-4bd6-96fa-cc82f650204e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:06.843025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38eb7f80-5ee1-4bd6-96fa-cc82f650204e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:06.844491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38eb7f80-5ee1-4bd6-96fa-cc82f650204e None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:06.859594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-38eb7f80-5ee1-4bd6-96fa-cc82f650204e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:44:06.859594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:06.859594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:06.859594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:06.859594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:06.859594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:06.859594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:06.859594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:06.859594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:06.859594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:06.859594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:06.859594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:06.859594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:44:06.859594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:44:06.859594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:44:06.859594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:44:06.860777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:06.860777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:06.860777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:06.860777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:06.860777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:06.860777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:06.860777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:06.860777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:06.860777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:06.860777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:06.860777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:06.860777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:06.860777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:06.860777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:06.860777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:06.860777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:06.860777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:44:06.861709 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:44:06.861709 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:44:06.861709 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:06.861709 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 347/690] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:44:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:06.861709 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38eb7f80-5ee1-4bd6-96fa-cc82f650204e None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:06.897116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520fbd8c-c427-409a-ac36-c7fd34968223 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:06.930981 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95092d29-1a79-4700-89d6-d6e1d5b8f3a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:06.931579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95092d29-1a79-4700-89d6-d6e1d5b8f3a0 None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:06.931889 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95092d29-1a79-4700-89d6-d6e1d5b8f3a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:06.932152 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95092d29-1a79-4700-89d6-d6e1d5b8f3a0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:06.932802 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95092d29-1a79-4700-89d6-d6e1d5b8f3a0 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:06.934159 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95092d29-1a79-4700-89d6-d6e1d5b8f3a0 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:06.967757 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 344/691] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:44:06 2022] POST /identity/v3/endpoints => generated 347 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:44:06.967757 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95092d29-1a79-4700-89d6-d6e1d5b8f3a0 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:06.978376 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38eb7f80-5ee1-4bd6-96fa-cc82f650204e None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:07.007689 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:07.026577 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c183dac6-9c0b-4760-a7bf-4f37865ebd66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:07.027520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c183dac6-9c0b-4760-a7bf-4f37865ebd66 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:07.027822 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c183dac6-9c0b-4760-a7bf-4f37865ebd66 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:07.028093 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c183dac6-9c0b-4760-a7bf-4f37865ebd66 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:07.028795 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c183dac6-9c0b-4760-a7bf-4f37865ebd66 None admin] RBAC: Adding query filter params (interface=public, service_id=3e3d7882e3aa4ef99f5b604c5cc12031, region_id=RegionOne) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:07.029133 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c183dac6-9c0b-4760-a7bf-4f37865ebd66 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:07.030473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c183dac6-9c0b-4760-a7bf-4f37865ebd66 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:07.037191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 348/692] 10.222.0.16 () {56 vars in 1251 bytes} [Sun Oct 30 00:44:06 2022] GET /identity/v3/endpoints?service_id=3e3d7882e3aa4ef99f5b604c5cc12031&interface=public®ion_id=RegionOne => generated 194 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:07.037726 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c183dac6-9c0b-4760-a7bf-4f37865ebd66 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:07.075431 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95092d29-1a79-4700-89d6-d6e1d5b8f3a0 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:07.092354 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:07.116839 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764f8bfa-0065-4ab9-8802-c71277524549 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:07.117485 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764f8bfa-0065-4ab9-8802-c71277524549 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:07.117742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764f8bfa-0065-4ab9-8802-c71277524549 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:07.118009 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764f8bfa-0065-4ab9-8802-c71277524549 None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:07.118658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764f8bfa-0065-4ab9-8802-c71277524549 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:07.118954 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764f8bfa-0065-4ab9-8802-c71277524549 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:07.120397 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764f8bfa-0065-4ab9-8802-c71277524549 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:07.134262 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 345/693] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:44:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:07.135416 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764f8bfa-0065-4ab9-8802-c71277524549 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:07.163480 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c183dac6-9c0b-4760-a7bf-4f37865ebd66 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:07.199479 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed305560-8d00-4684-8d7c-8bab23f17fec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:07.200485 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed305560-8d00-4684-8d7c-8bab23f17fec None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:07.200903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed305560-8d00-4684-8d7c-8bab23f17fec None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:07.201311 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed305560-8d00-4684-8d7c-8bab23f17fec None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:07.202155 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed305560-8d00-4684-8d7c-8bab23f17fec None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:07.203787 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed305560-8d00-4684-8d7c-8bab23f17fec None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:07.214180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-ed305560-8d00-4684-8d7c-8bab23f17fec None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 30 00:44:07.214180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:07.214180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:44:07.214180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:44:07.214180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:44:07.214180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:44:07.214180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:44:07.214180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:07.214180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 30 00:44:07.214180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:07.214180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:44:07.214180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:07.214180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:07.214180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:07.214180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:07.214180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:07.214180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:07.215500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:07.215500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:07.215500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:07.215500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:07.215500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:07.215500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:07.215500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:44:07.215500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:44:07.215500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:44:07.215500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:44:07.215500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:07.215500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:07.215500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:07.215500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:07.215500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:07.215500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:07.216443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:07.216443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:07.216443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:07.216443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:07.216443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:07.216443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:07.216443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:07.216443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:07.216443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:07.216443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:07.216443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:44:07.216443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:07.216443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 30 00:44:07.216443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:07.217545 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 349/694] 10.222.0.16 () {56 vars in 1114 bytes} [Sun Oct 30 00:44:07 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:07.218370 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed305560-8d00-4684-8d7c-8bab23f17fec None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:07.246287 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764f8bfa-0065-4ab9-8802-c71277524549 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:07.248696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:07.284608 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035efb21-532b-4dd5-8b83-3930eff140c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:07.285215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035efb21-532b-4dd5-8b83-3930eff140c5 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:07.285487 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035efb21-532b-4dd5-8b83-3930eff140c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:07.285751 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035efb21-532b-4dd5-8b83-3930eff140c5 None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:07.286442 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035efb21-532b-4dd5-8b83-3930eff140c5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:07.287948 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035efb21-532b-4dd5-8b83-3930eff140c5 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:07.304128 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 346/695] 10.222.0.16 () {56 vars in 1087 bytes} [Sun Oct 30 00:44:07 2022] GET /identity/v3/services => generated 1497 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:44:07.304964 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035efb21-532b-4dd5-8b83-3930eff140c5 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:07.324763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed305560-8d00-4684-8d7c-8bab23f17fec None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:07.337457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:07.380523 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695bc109-3277-4373-bca1-f01268aa912b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:07.381170 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695bc109-3277-4373-bca1-f01268aa912b None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:07.381439 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695bc109-3277-4373-bca1-f01268aa912b None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:07.381706 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695bc109-3277-4373-bca1-f01268aa912b None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:07.394276 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695bc109-3277-4373-bca1-f01268aa912b None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:07.396791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695bc109-3277-4373-bca1-f01268aa912b None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:07.416549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-695bc109-3277-4373-bca1-f01268aa912b None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 30 00:44:07.416549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:07.416549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:07.416549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:07.416549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:07.416549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:07.416549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:07.416549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:07.416549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:07.416549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:07.416549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:07.416549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:07.416549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:44:07.416549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:44:07.416549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:44:07.416549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:44:07.417583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:07.417583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:07.417583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:44:07.417583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:44:07.417583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:44:07.417583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:44:07.417583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:07.417583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:07.417583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:07.417583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:07.417583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:07.417583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:07.417583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:07.417583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:07.417583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:07.417583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:07.417583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:07.417583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:07.418510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:07.418510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:07.418510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:44:07.418510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:44:07.418510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:44:07.418510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:44:07.418510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:44:07.418510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:44:07.418510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 30 00:44:07.418510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:07.418510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 350/696] 10.222.0.16 () {56 vars in 1117 bytes} [Sun Oct 30 00:44:07 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:07.427353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695bc109-3277-4373-bca1-f01268aa912b None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:07.464751 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035efb21-532b-4dd5-8b83-3930eff140c5 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:07.514463 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e79d974-0afa-42ba-b2fd-8b2bbbd3f52a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:07.515081 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e79d974-0afa-42ba-b2fd-8b2bbbd3f52a None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:07.515356 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e79d974-0afa-42ba-b2fd-8b2bbbd3f52a None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:07.515623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e79d974-0afa-42ba-b2fd-8b2bbbd3f52a None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:07.516339 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e79d974-0afa-42ba-b2fd-8b2bbbd3f52a None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:07.516637 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e79d974-0afa-42ba-b2fd-8b2bbbd3f52a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:07.518068 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e79d974-0afa-42ba-b2fd-8b2bbbd3f52a None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:07.523669 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 347/697] 10.222.0.16 () {56 vars in 1114 bytes} [Sun Oct 30 00:44:07 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:07.529208 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e79d974-0afa-42ba-b2fd-8b2bbbd3f52a None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:07.556379 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695bc109-3277-4373-bca1-f01268aa912b None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:07.571581 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:07.592462 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d533e7-1656-43cf-88cc-7767ddad37f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:07.593072 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d533e7-1656-43cf-88cc-7767ddad37f6 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:07.593342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d533e7-1656-43cf-88cc-7767ddad37f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:07.593607 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d533e7-1656-43cf-88cc-7767ddad37f6 None admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:07.594422 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d533e7-1656-43cf-88cc-7767ddad37f6 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:07.594729 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d533e7-1656-43cf-88cc-7767ddad37f6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:07.596330 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d533e7-1656-43cf-88cc-7767ddad37f6 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:07.614902 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 351/698] 10.222.0.16 () {56 vars in 1149 bytes} [Sun Oct 30 00:44:07 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:07.620922 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d533e7-1656-43cf-88cc-7767ddad37f6 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:07.650796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:07.662263 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e79d974-0afa-42ba-b2fd-8b2bbbd3f52a None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:07.706907 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d23446-d894-4cbd-bda1-3b9c1c6e6d27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:07.707528 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d23446-d894-4cbd-bda1-3b9c1c6e6d27 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:07.707836 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d23446-d894-4cbd-bda1-3b9c1c6e6d27 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:07.708504 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d23446-d894-4cbd-bda1-3b9c1c6e6d27 None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:07.709275 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d23446-d894-4cbd-bda1-3b9c1c6e6d27 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:07.709717 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d23446-d894-4cbd-bda1-3b9c1c6e6d27 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:07.711393 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d23446-d894-4cbd-bda1-3b9c1c6e6d27 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:07.718179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 348/699] 10.222.0.16 () {56 vars in 1114 bytes} [Sun Oct 30 00:44:07 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:44:07.724500 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d23446-d894-4cbd-bda1-3b9c1c6e6d27 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:07.739128 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d533e7-1656-43cf-88cc-7767ddad37f6 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:07.783127 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646d4f90-1ed4-44da-81c4-494b0a9b8f55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:07.783871 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646d4f90-1ed4-44da-81c4-494b0a9b8f55 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:07.784357 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646d4f90-1ed4-44da-81c4-494b0a9b8f55 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:07.784633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646d4f90-1ed4-44da-81c4-494b0a9b8f55 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:07.820820 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:07.829204 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d23446-d894-4cbd-bda1-3b9c1c6e6d27 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:07.870864 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646d4f90-1ed4-44da-81c4-494b0a9b8f55 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:07.872349 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646d4f90-1ed4-44da-81c4-494b0a9b8f55 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:07.874935 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f85754-7097-4270-bf1a-86556f98f3be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:07.876486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f85754-7097-4270-bf1a-86556f98f3be None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:07.876784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f85754-7097-4270-bf1a-86556f98f3be None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:07.877061 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f85754-7097-4270-bf1a-86556f98f3be None admin] PATH_INFO: `/v3/endpoints` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:07.877980 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f85754-7097-4270-bf1a-86556f98f3be None admin] RBAC: Adding query filter params (interface=public, service_id=5d6610792ec94205b591759e55a3131e, region_id=RegionOne) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:07.878286 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f85754-7097-4270-bf1a-86556f98f3be None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:07.879694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f85754-7097-4270-bf1a-86556f98f3be None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:07.889466 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 349/700] 10.222.0.16 () {56 vars in 1251 bytes} [Sun Oct 30 00:44:07 2022] GET /identity/v3/endpoints?service_id=5d6610792ec94205b591759e55a3131e&interface=public®ion_id=RegionOne => generated 194 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:07.894974 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f85754-7097-4270-bf1a-86556f98f3be None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:07.913306 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:08.006388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f85754-7097-4270-bf1a-86556f98f3be None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:08.010473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 352/701] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:44:07 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 30 00:44:08.011398 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646d4f90-1ed4-44da-81c4-494b0a9b8f55 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:08.052248 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381825ae-bd92-49ec-a634-edd072f12e45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:08.052859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381825ae-bd92-49ec-a634-edd072f12e45 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:08.053134 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381825ae-bd92-49ec-a634-edd072f12e45 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:08.053402 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381825ae-bd92-49ec-a634-edd072f12e45 None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:08.054070 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381825ae-bd92-49ec-a634-edd072f12e45 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:08.055464 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381825ae-bd92-49ec-a634-edd072f12e45 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:08.068204 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 350/702] 10.222.0.16 () {56 vars in 1087 bytes} [Sun Oct 30 00:44:07 2022] GET /identity/v3/services => generated 1497 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 00:44:08.126594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646d4f90-1ed4-44da-81c4-494b0a9b8f55 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:08.157614 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:08.168669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3d8ab7-bba7-4f7d-87da-e66b3ae7d98c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:08.169673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3d8ab7-bba7-4f7d-87da-e66b3ae7d98c None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:08.170137 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3d8ab7-bba7-4f7d-87da-e66b3ae7d98c None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:08.170589 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3d8ab7-bba7-4f7d-87da-e66b3ae7d98c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:08.182249 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3d8ab7-bba7-4f7d-87da-e66b3ae7d98c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:08.184290 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3d8ab7-bba7-4f7d-87da-e66b3ae7d98c None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:08.202208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-ff3d8ab7-bba7-4f7d-87da-e66b3ae7d98c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:44:08.202208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:08.202208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:08.202208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:08.202208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:08.202208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:08.202208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:08.202208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:08.202208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:08.202208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:08.202208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:08.202208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:08.202208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:44:08.202208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:44:08.202208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:44:08.202208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:44:08.203235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:08.203235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:08.203235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:08.203235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:08.203235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:08.203235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:08.203235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:08.203235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:08.203235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:08.203235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:08.203235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:08.203235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:08.203235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:08.203235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:08.203235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:08.203235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:08.203235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:44:08.204170 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:44:08.204170 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:44:08.204170 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:08.206132 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 353/703] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:44:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:08.212784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381825ae-bd92-49ec-a634-edd072f12e45 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:08.243274 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:08.331983 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381825ae-bd92-49ec-a634-edd072f12e45 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:08.371757 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0708e6-393b-4d4c-a2ce-406f0ac28484 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:08.371757 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0708e6-393b-4d4c-a2ce-406f0ac28484 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:08.371757 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0708e6-393b-4d4c-a2ce-406f0ac28484 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:08.371757 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0708e6-393b-4d4c-a2ce-406f0ac28484 None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:08.372160 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0708e6-393b-4d4c-a2ce-406f0ac28484 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:08.372160 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0708e6-393b-4d4c-a2ce-406f0ac28484 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:08.372516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0708e6-393b-4d4c-a2ce-406f0ac28484 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:08.384846 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 351/704] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:44:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:08.390516 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3d8ab7-bba7-4f7d-87da-e66b3ae7d98c None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:08.495349 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3d8ab7-bba7-4f7d-87da-e66b3ae7d98c None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:08.529275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39cac612-3fcd-4da2-8912-19d6845d9972 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:08.529907 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cac612-3fcd-4da2-8912-19d6845d9972 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:08.530176 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cac612-3fcd-4da2-8912-19d6845d9972 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:08.530442 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cac612-3fcd-4da2-8912-19d6845d9972 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:08.537660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39cac612-3fcd-4da2-8912-19d6845d9972 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:08.539170 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39cac612-3fcd-4da2-8912-19d6845d9972 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:08.550478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-39cac612-3fcd-4da2-8912-19d6845d9972 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:44:08.550478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:08.550478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:08.550478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:08.550478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:08.550478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:08.550478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:08.550478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:08.550478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:08.550478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:08.550478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:08.550478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:08.550478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:44:08.550478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:44:08.550478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:44:08.551392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:44:08.551392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:08.551392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:08.551392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:08.551392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:08.551392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:08.551392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:08.551392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:08.551392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:08.551392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:08.551392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:08.551392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:08.551392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:08.551392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:08.551392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:08.551392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:08.551392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:08.552342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:44:08.552342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:44:08.552342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:44:08.552342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:44:08.552342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:44:08.552342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:44:08.552342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:44:08.552342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:08.552342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 354/705] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:44:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:08.558409 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0708e6-393b-4d4c-a2ce-406f0ac28484 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:08.649943 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:08.666971 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0708e6-393b-4d4c-a2ce-406f0ac28484 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:08.703054 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca7f35e-4c54-40a0-b97e-1dd62a22a5f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:08.703660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca7f35e-4c54-40a0-b97e-1dd62a22a5f2 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:08.703962 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca7f35e-4c54-40a0-b97e-1dd62a22a5f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:08.704224 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca7f35e-4c54-40a0-b97e-1dd62a22a5f2 None admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:08.704959 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca7f35e-4c54-40a0-b97e-1dd62a22a5f2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:08.705255 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca7f35e-4c54-40a0-b97e-1dd62a22a5f2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:08.706783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca7f35e-4c54-40a0-b97e-1dd62a22a5f2 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:08.719621 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 352/706] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:44:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:08.725939 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39cac612-3fcd-4da2-8912-19d6845d9972 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:08.767068 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:08.828596 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39cac612-3fcd-4da2-8912-19d6845d9972 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:08.862566 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbd5a0e-21f4-41a2-9d69-f6f297f10f24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:08.863170 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbd5a0e-21f4-41a2-9d69-f6f297f10f24 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:08.863440 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbd5a0e-21f4-41a2-9d69-f6f297f10f24 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:08.863702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbd5a0e-21f4-41a2-9d69-f6f297f10f24 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:08.864442 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efbd5a0e-21f4-41a2-9d69-f6f297f10f24 None admin] RBAC: Adding query filter params (role.id=2aa395773c0a4706897ec0992f3a769a, scope.project.id=658799ed31d2472fb36af4aff0f03aab, user.id=4716e7afe2784e2a9b872996d895c374) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:08.864739 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efbd5a0e-21f4-41a2-9d69-f6f297f10f24 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:08.866157 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efbd5a0e-21f4-41a2-9d69-f6f297f10f24 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:08.872555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 355/707] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:44:08 2022] GET /identity/v3/role_assignments?user.id=4716e7afe2784e2a9b872996d895c374&scope.project.id=658799ed31d2472fb36af4aff0f03aab&role.id=2aa395773c0a4706897ec0992f3a769a => generated 608 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:09.159981 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817906c0-f52a-4f71-9ca1-7f08859b969e None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:09.160370 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817906c0-f52a-4f71-9ca1-7f08859b969e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:09.160549 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817906c0-f52a-4f71-9ca1-7f08859b969e None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:09.161989 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 353/708] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:09.167497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5216f3dc-f3d6-4793-9e93-2214cd0910be None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:09.168027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5216f3dc-f3d6-4793-9e93-2214cd0910be None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:09.168495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5216f3dc-f3d6-4793-9e93-2214cd0910be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:09.187165 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-5216f3dc-f3d6-4793-9e93-2214cd0910be None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:09.188732 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-5216f3dc-f3d6-4793-9e93-2214cd0910be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:09.263100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 356/709] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:09 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:09.526837 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817906c0-f52a-4f71-9ca1-7f08859b969e None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:09.571497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee5b359-5217-426f-88e1-cd2c0c0582e1 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:09.571860 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee5b359-5217-426f-88e1-cd2c0c0582e1 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:09.572078 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee5b359-5217-426f-88e1-cd2c0c0582e1 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:09.573642 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 357/710] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:09.578871 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23640abf-6113-48d0-8ce7-a5314ea2220a None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:09.579150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23640abf-6113-48d0-8ce7-a5314ea2220a None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:09.579417 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23640abf-6113-48d0-8ce7-a5314ea2220a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:09.597721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-23640abf-6113-48d0-8ce7-a5314ea2220a None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:09.598901 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-23640abf-6113-48d0-8ce7-a5314ea2220a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:09.637467 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:09.640031 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817906c0-f52a-4f71-9ca1-7f08859b969e None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:09.676033 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4339f862-0502-4c58-aeaa-24c348b78bf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:09.676772 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4339f862-0502-4c58-aeaa-24c348b78bf6 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:09.677058 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4339f862-0502-4c58-aeaa-24c348b78bf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:09.677326 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4339f862-0502-4c58-aeaa-24c348b78bf6 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:09.677899 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4339f862-0502-4c58-aeaa-24c348b78bf6 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:09.679266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4339f862-0502-4c58-aeaa-24c348b78bf6 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:09.682076 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 358/711] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:09 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 30 00:44:09.696336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-4339f862-0502-4c58-aeaa-24c348b78bf6 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 30 00:44:09.696336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:09.696336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:44:09.696336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:44:09.696336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:44:09.696336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:44:09.696336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:44:09.696336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:09.696336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 30 00:44:09.696336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:09.696336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:44:09.696336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:09.696336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:09.696336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:09.696336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:09.696336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:09.696336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:09.697635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:09.697635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:09.697635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:09.697635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:09.697635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:09.697635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:09.697635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:44:09.697635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:44:09.697635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:44:09.697635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:44:09.697635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:09.697635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:09.697635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:09.697635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:09.697635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:09.697635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:09.698536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:09.698536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:09.698536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:09.698536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:09.698536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:09.698536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:09.698536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:09.698536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:09.698536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:09.698536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:09.698536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:44:09.698536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:09.698536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 30 00:44:09.698536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:09.700756 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 354/712] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:44:09 2022] GET /identity/v3/services/compute => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:09.707184 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23640abf-6113-48d0-8ce7-a5314ea2220a None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:09.805671 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23640abf-6113-48d0-8ce7-a5314ea2220a None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:09.842588 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e873659-43dd-46ee-8cf9-ac4d2a3adf80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:09.843473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e873659-43dd-46ee-8cf9-ac4d2a3adf80 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:09.843916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e873659-43dd-46ee-8cf9-ac4d2a3adf80 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:09.844355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e873659-43dd-46ee-8cf9-ac4d2a3adf80 None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:09.845142 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e873659-43dd-46ee-8cf9-ac4d2a3adf80 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:09.845647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e873659-43dd-46ee-8cf9-ac4d2a3adf80 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:09.847170 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e873659-43dd-46ee-8cf9-ac4d2a3adf80 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:09.852982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 359/713] 10.222.0.16 () {56 vars in 1112 bytes} [Sun Oct 30 00:44:09 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:09.858744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4339f862-0502-4c58-aeaa-24c348b78bf6 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:09.871156 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:09.957996 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e873659-43dd-46ee-8cf9-ac4d2a3adf80 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:09.966464 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4339f862-0502-4c58-aeaa-24c348b78bf6 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:09.987985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:10.008627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188545b8-16ec-4c35-a4e1-ecdd12531bf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:10.009234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188545b8-16ec-4c35-a4e1-ecdd12531bf0 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:10.009565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188545b8-16ec-4c35-a4e1-ecdd12531bf0 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:10.009844 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188545b8-16ec-4c35-a4e1-ecdd12531bf0 None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:10.010473 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188545b8-16ec-4c35-a4e1-ecdd12531bf0 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:10.010777 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188545b8-16ec-4c35-a4e1-ecdd12531bf0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:10.012859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188545b8-16ec-4c35-a4e1-ecdd12531bf0 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:10.019390 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 355/714] 10.222.0.16 () {56 vars in 1112 bytes} [Sun Oct 30 00:44:09 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:10.085155 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e873659-43dd-46ee-8cf9-ac4d2a3adf80 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:10.123663 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a37bfe-a47b-4745-9870-44085b9083ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:10.124751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a37bfe-a47b-4745-9870-44085b9083ce None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:10.125328 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a37bfe-a47b-4745-9870-44085b9083ce None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:10.125764 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a37bfe-a47b-4745-9870-44085b9083ce None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:10.126480 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a37bfe-a47b-4745-9870-44085b9083ce None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:10.128373 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a37bfe-a47b-4745-9870-44085b9083ce None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:10.138162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-21a37bfe-a47b-4745-9870-44085b9083ce None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 30 00:44:10.138162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:10.138162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:44:10.138162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:44:10.138162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:44:10.138162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:44:10.138162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:44:10.138162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:10.138162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 30 00:44:10.138162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:10.138162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:44:10.138162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:10.138162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:10.138162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:10.138162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:10.138162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:10.138162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:10.139410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:10.139410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:10.139410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:10.139410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:10.139410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:10.139410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:10.139410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:44:10.139410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:44:10.139410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:44:10.139410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:44:10.139410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:10.139410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:10.139410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:10.139410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:10.139410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:10.139410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:10.140455 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:10.140455 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:10.140455 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:10.140455 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:10.140455 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:10.140455 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:10.140455 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:10.140455 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:10.140455 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:10.140455 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:10.140455 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:44:10.140455 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:10.140455 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 30 00:44:10.140455 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:10.143094 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 360/715] 10.222.0.16 () {56 vars in 1126 bytes} [Sun Oct 30 00:44:09 2022] GET /identity/v3/services/object-store => generated 93 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:44:10.149333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188545b8-16ec-4c35-a4e1-ecdd12531bf0 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:10.230115 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:10.266043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188545b8-16ec-4c35-a4e1-ecdd12531bf0 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:10.301624 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b49806-29a8-4b0f-b2a1-164ace53432c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:10.302240 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b49806-29a8-4b0f-b2a1-164ace53432c None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:10.302549 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b49806-29a8-4b0f-b2a1-164ace53432c None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:10.302824 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b49806-29a8-4b0f-b2a1-164ace53432c None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:10.303450 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b49806-29a8-4b0f-b2a1-164ace53432c None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:10.303764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b49806-29a8-4b0f-b2a1-164ace53432c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:10.305126 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b49806-29a8-4b0f-b2a1-164ace53432c None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:10.310290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 356/716] 10.222.0.16 () {56 vars in 1122 bytes} [Sun Oct 30 00:44:10 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:10.316298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a37bfe-a47b-4745-9870-44085b9083ce None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:10.370097 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a385f8-6a53-4291-8456-e563562111a3 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:10.370405 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a385f8-6a53-4291-8456-e563562111a3 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:10.370641 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a385f8-6a53-4291-8456-e563562111a3 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:10.374305 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 357/717] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:44:10.377743 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2659e9-b3e3-45ca-ac91-dbd0d6589448 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:10.377990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2659e9-b3e3-45ca-ac91-dbd0d6589448 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:10.378273 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2659e9-b3e3-45ca-ac91-dbd0d6589448 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:10.401716 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-eb2659e9-b3e3-45ca-ac91-dbd0d6589448 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:10.403172 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-eb2659e9-b3e3-45ca-ac91-dbd0d6589448 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:10.414779 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:10.430586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a37bfe-a47b-4745-9870-44085b9083ce None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:10.467276 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818a37e2-1c4e-4da3-ac4a-8c0d1dcf0eb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:10.467948 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818a37e2-1c4e-4da3-ac4a-8c0d1dcf0eb1 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:10.468226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818a37e2-1c4e-4da3-ac4a-8c0d1dcf0eb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:10.468495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818a37e2-1c4e-4da3-ac4a-8c0d1dcf0eb1 None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:10.469116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818a37e2-1c4e-4da3-ac4a-8c0d1dcf0eb1 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:10.469411 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818a37e2-1c4e-4da3-ac4a-8c0d1dcf0eb1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:10.471014 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818a37e2-1c4e-4da3-ac4a-8c0d1dcf0eb1 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:10.476678 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 361/718] 10.222.0.16 () {56 vars in 1122 bytes} [Sun Oct 30 00:44:10 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:10.481535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 358/719] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:10 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 30 00:44:10.481846 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818a37e2-1c4e-4da3-ac4a-8c0d1dcf0eb1 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:10.590915 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818a37e2-1c4e-4da3-ac4a-8c0d1dcf0eb1 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:10.625305 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b7453a-7d1f-4b1b-b7f2-e5899794bb99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:10.626191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b7453a-7d1f-4b1b-b7f2-e5899794bb99 None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:10.626657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b7453a-7d1f-4b1b-b7f2-e5899794bb99 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:10.627355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b7453a-7d1f-4b1b-b7f2-e5899794bb99 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:10.628200 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b7453a-7d1f-4b1b-b7f2-e5899794bb99 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:10.629728 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b7453a-7d1f-4b1b-b7f2-e5899794bb99 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:10.653179 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 362/720] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:44:10 2022] POST /identity/v3/endpoints => generated 349 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:44:10.771473 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2659e9-b3e3-45ca-ac91-dbd0d6589448 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:10.844723 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:10.881099 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2659e9-b3e3-45ca-ac91-dbd0d6589448 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:10.922379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c69be0-924c-4b1c-9b22-384d3555e072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:10.923005 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c69be0-924c-4b1c-9b22-384d3555e072 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:10.923273 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c69be0-924c-4b1c-9b22-384d3555e072 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:10.923598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c69be0-924c-4b1c-9b22-384d3555e072 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:10.924256 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c69be0-924c-4b1c-9b22-384d3555e072 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:10.925654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c69be0-924c-4b1c-9b22-384d3555e072 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:10.934696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-64c69be0-924c-4b1c-9b22-384d3555e072 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 30 00:44:10.934696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:10.934696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:44:10.934696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:44:10.934696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:44:10.934696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:44:10.934696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:44:10.934696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:10.934696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 30 00:44:10.934696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:10.934696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:44:10.934696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:10.934696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:10.934696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:10.934696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:10.934696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:10.934696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:10.935765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:10.935765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:10.935765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:10.935765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:10.935765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:10.935765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:10.935765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:44:10.935765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:44:10.935765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:44:10.935765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:44:10.935765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:10.935765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:10.935765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:10.935765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:10.935765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:10.935765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:10.936973 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:10.936973 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:10.936973 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:10.936973 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:10.936973 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:10.936973 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:10.936973 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:10.936973 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:10.936973 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:10.936973 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:10.936973 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:44:10.936973 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:10.936973 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 30 00:44:10.936973 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:10.936973 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 359/721] 10.222.0.16 () {56 vars in 1114 bytes} [Sun Oct 30 00:44:10 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:10.942807 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b7453a-7d1f-4b1b-b7f2-e5899794bb99 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:10.957207 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:11.058863 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b7453a-7d1f-4b1b-b7f2-e5899794bb99 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:11.093633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7e6e93-96b6-4768-95a0-70056a1a88a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:11.094248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7e6e93-96b6-4768-95a0-70056a1a88a0 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:11.094763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7e6e93-96b6-4768-95a0-70056a1a88a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:11.095027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7e6e93-96b6-4768-95a0-70056a1a88a0 None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:11.095700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7e6e93-96b6-4768-95a0-70056a1a88a0 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:11.096012 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7e6e93-96b6-4768-95a0-70056a1a88a0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:11.097451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7e6e93-96b6-4768-95a0-70056a1a88a0 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:11.102538 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 363/722] 10.222.0.16 () {56 vars in 1114 bytes} [Sun Oct 30 00:44:10 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:11.108178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c69be0-924c-4b1c-9b22-384d3555e072 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:11.145256 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403d1ee3-b97e-4c37-a60e-50f60c5b1d0e None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:11.145433 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403d1ee3-b97e-4c37-a60e-50f60c5b1d0e None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:11.145704 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403d1ee3-b97e-4c37-a60e-50f60c5b1d0e None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:11.147280 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 364/723] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:44:11.152195 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6ca89a-1a33-4242-9ac7-90af7bcf05b8 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:11.152473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6ca89a-1a33-4242-9ac7-90af7bcf05b8 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:11.152741 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6ca89a-1a33-4242-9ac7-90af7bcf05b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:11.170490 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-ef6ca89a-1a33-4242-9ac7-90af7bcf05b8 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:11.171665 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ef6ca89a-1a33-4242-9ac7-90af7bcf05b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:11.215452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c69be0-924c-4b1c-9b22-384d3555e072 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:11.237550 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 365/724] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:11 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 30 00:44:11.243825 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:11.253433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bf2eb9-1ca5-4019-a821-33e348a21042 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:11.254336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bf2eb9-1ca5-4019-a821-33e348a21042 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:11.254807 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bf2eb9-1ca5-4019-a821-33e348a21042 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:11.255263 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bf2eb9-1ca5-4019-a821-33e348a21042 None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:11.256125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bf2eb9-1ca5-4019-a821-33e348a21042 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:11.256617 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bf2eb9-1ca5-4019-a821-33e348a21042 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:11.258201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bf2eb9-1ca5-4019-a821-33e348a21042 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:11.269081 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 360/725] 10.222.0.16 () {56 vars in 1114 bytes} [Sun Oct 30 00:44:11 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:11.270646 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6ca89a-1a33-4242-9ac7-90af7bcf05b8 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:11.274506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:11.376444 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6ca89a-1a33-4242-9ac7-90af7bcf05b8 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:11.410800 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25110695-c1e0-4bcc-b84e-f43cc4ac00fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:11.411323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25110695-c1e0-4bcc-b84e-f43cc4ac00fa None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:11.411593 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25110695-c1e0-4bcc-b84e-f43cc4ac00fa None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:11.411881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25110695-c1e0-4bcc-b84e-f43cc4ac00fa None admin] PATH_INFO: `/v3/endpoints` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:11.412547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25110695-c1e0-4bcc-b84e-f43cc4ac00fa None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:11.413902 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25110695-c1e0-4bcc-b84e-f43cc4ac00fa None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:11.435229 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 366/726] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:44:11 2022] POST /identity/v3/endpoints => generated 346 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:44:11.506156 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bf2eb9-1ca5-4019-a821-33e348a21042 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:11.620040 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bf2eb9-1ca5-4019-a821-33e348a21042 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:11.661376 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885388e4-8497-4cef-bc61-8d6f122b639e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:11.662268 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885388e4-8497-4cef-bc61-8d6f122b639e None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:11.662545 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885388e4-8497-4cef-bc61-8d6f122b639e None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:11.662809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885388e4-8497-4cef-bc61-8d6f122b639e None admin] PATH_INFO: `/v3/services/image` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:11.663387 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885388e4-8497-4cef-bc61-8d6f122b639e None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:11.664839 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885388e4-8497-4cef-bc61-8d6f122b639e None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:11.674575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-885388e4-8497-4cef-bc61-8d6f122b639e None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 30 00:44:11.674575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:11.674575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:44:11.674575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:44:11.674575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:44:11.674575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:44:11.674575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:44:11.674575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:11.674575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 30 00:44:11.674575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:11.674575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:44:11.674575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:11.674575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:11.674575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:11.674575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:11.674575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:11.674575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:11.675654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:11.675654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:11.675654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:11.675654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:11.675654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:11.675654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:11.675654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:44:11.675654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:44:11.675654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:44:11.675654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:44:11.675654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:11.675654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:11.675654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:11.675654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:11.675654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:11.675654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:11.677139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:11.677139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:11.677139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:11.677139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:11.677139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:11.677139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:11.677139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:11.677139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:11.677139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:11.677139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:11.677139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:44:11.677139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:11.677139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 30 00:44:11.677139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:11.677139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 361/727] 10.222.0.16 () {56 vars in 1105 bytes} [Sun Oct 30 00:44:11 2022] GET /identity/v3/services/image => generated 86 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:11.681763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25110695-c1e0-4bcc-b84e-f43cc4ac00fa None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:11.748189 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:11.791336 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25110695-c1e0-4bcc-b84e-f43cc4ac00fa None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:11.826068 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39283740-8f8c-4780-8101-d8ef54982c1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:11.827036 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39283740-8f8c-4780-8101-d8ef54982c1e None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:11.827526 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39283740-8f8c-4780-8101-d8ef54982c1e None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:11.828011 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39283740-8f8c-4780-8101-d8ef54982c1e None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:11.828817 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39283740-8f8c-4780-8101-d8ef54982c1e None admin] RBAC: Adding query filter params (name=image) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:11.829301 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39283740-8f8c-4780-8101-d8ef54982c1e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:11.830863 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39283740-8f8c-4780-8101-d8ef54982c1e None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:11.837719 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 367/728] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:44:11 2022] GET /identity/v3/services?name=image => generated 122 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:11.843572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885388e4-8497-4cef-bc61-8d6f122b639e None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:11.899822 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:11.952471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885388e4-8497-4cef-bc61-8d6f122b639e None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:11.991992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae39d62e-8056-4e95-b87d-00bb3748421e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:11.992601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae39d62e-8056-4e95-b87d-00bb3748421e None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:11.992872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae39d62e-8056-4e95-b87d-00bb3748421e None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:11.993178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae39d62e-8056-4e95-b87d-00bb3748421e None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:11.993768 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae39d62e-8056-4e95-b87d-00bb3748421e None admin] RBAC: Adding query filter params (type=image) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:11.994064 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae39d62e-8056-4e95-b87d-00bb3748421e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:11.995478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae39d62e-8056-4e95-b87d-00bb3748421e None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:12.000774 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 362/729] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:44:11 2022] GET /identity/v3/services?type=image => generated 122 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:12.372969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24800097-1659-412b-a0e6-35ea2af11ca8 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:12.373469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24800097-1659-412b-a0e6-35ea2af11ca8 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:12.373563 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24800097-1659-412b-a0e6-35ea2af11ca8 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:12.375293 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 368/730] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:12.381029 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1b5470-70a3-4228-a75b-a62452ca0ddf None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:12.381313 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1b5470-70a3-4228-a75b-a62452ca0ddf None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:12.381589 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1b5470-70a3-4228-a75b-a62452ca0ddf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:12.400236 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-0e1b5470-70a3-4228-a75b-a62452ca0ddf None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:12.401417 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-0e1b5470-70a3-4228-a75b-a62452ca0ddf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:12.473923 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 363/731] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:12 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:12.746341 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24800097-1659-412b-a0e6-35ea2af11ca8 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:12.856338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24800097-1659-412b-a0e6-35ea2af11ca8 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:12.882984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:12.895300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b09e77-13fe-4c86-962d-9b0d0ef421a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:12.896243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b09e77-13fe-4c86-962d-9b0d0ef421a1 None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:12.896708 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b09e77-13fe-4c86-962d-9b0d0ef421a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:12.897175 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b09e77-13fe-4c86-962d-9b0d0ef421a1 None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:12.898009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b09e77-13fe-4c86-962d-9b0d0ef421a1 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:12.899625 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b09e77-13fe-4c86-962d-9b0d0ef421a1 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:12.919368 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 369/732] 10.222.0.16 () {60 vars in 1141 bytes} [Sun Oct 30 00:44:12 2022] POST /identity/v3/services => generated 242 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:44:12.947386 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2963d862-815f-4f65-9e10-90d8e7248ba5 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:12.947963 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2963d862-815f-4f65-9e10-90d8e7248ba5 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:12.947963 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2963d862-815f-4f65-9e10-90d8e7248ba5 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:12.949664 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 364/733] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:12.955610 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb828ad-1a0f-4a69-812e-9767365bac17 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:12.955965 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb828ad-1a0f-4a69-812e-9767365bac17 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:12.956287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb828ad-1a0f-4a69-812e-9767365bac17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:12.975290 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-ffb828ad-1a0f-4a69-812e-9767365bac17 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:12.976531 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ffb828ad-1a0f-4a69-812e-9767365bac17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:13.057685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 370/734] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:12 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:13.325724 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2963d862-815f-4f65-9e10-90d8e7248ba5 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:13.385481 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:13.442710 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2963d862-815f-4f65-9e10-90d8e7248ba5 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:13.482007 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc4caea-838f-43fd-8655-ff8e60077cbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:13.482685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc4caea-838f-43fd-8655-ff8e60077cbe None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:13.482963 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc4caea-838f-43fd-8655-ff8e60077cbe None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:13.483231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc4caea-838f-43fd-8655-ff8e60077cbe None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:13.483844 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc4caea-838f-43fd-8655-ff8e60077cbe None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:13.485201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc4caea-838f-43fd-8655-ff8e60077cbe None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:13.494727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-2dc4caea-838f-43fd-8655-ff8e60077cbe None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 30 00:44:13.494727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:13.494727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:44:13.494727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:44:13.494727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:44:13.494727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:44:13.494727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:44:13.494727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:13.494727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 30 00:44:13.494727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:13.494727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:44:13.494727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:13.494727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:13.494727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:13.494727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:13.494727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:13.494727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:13.495754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:13.495754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:13.495754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:13.495754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:13.495754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:13.495754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:13.495754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:44:13.495754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:44:13.495754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:44:13.495754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:44:13.495754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:13.495754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:13.495754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:13.495754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:13.495754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:13.495754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:13.496600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:13.496600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:13.496600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:13.496600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:13.496600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:13.496600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:13.496600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:13.496600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:13.496600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:13.496600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:13.496600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:44:13.496600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:13.496600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 30 00:44:13.496600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:13.496600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 365/735] 10.222.0.16 () {56 vars in 1126 bytes} [Sun Oct 30 00:44:13 2022] GET /identity/v3/services/object-store => generated 93 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:13.502378 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb828ad-1a0f-4a69-812e-9767365bac17 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:13.599271 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb828ad-1a0f-4a69-812e-9767365bac17 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:13.610474 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:13.635475 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dce78ef-3634-412b-92f9-598db92c5b5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:13.636112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dce78ef-3634-412b-92f9-598db92c5b5d None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:13.636388 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dce78ef-3634-412b-92f9-598db92c5b5d None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:13.636654 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dce78ef-3634-412b-92f9-598db92c5b5d None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:13.637276 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dce78ef-3634-412b-92f9-598db92c5b5d None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:13.637573 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dce78ef-3634-412b-92f9-598db92c5b5d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:13.638921 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dce78ef-3634-412b-92f9-598db92c5b5d None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:13.644451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 371/736] 10.222.0.16 () {56 vars in 1122 bytes} [Sun Oct 30 00:44:13 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:13.650223 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc4caea-838f-43fd-8655-ff8e60077cbe None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:13.761127 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc4caea-838f-43fd-8655-ff8e60077cbe None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:13.770478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:13.795650 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e71a77-304e-43e8-bbf4-f6d248d469b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:13.796288 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e71a77-304e-43e8-bbf4-f6d248d469b0 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:13.796561 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e71a77-304e-43e8-bbf4-f6d248d469b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:13.796827 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e71a77-304e-43e8-bbf4-f6d248d469b0 None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:13.797454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e71a77-304e-43e8-bbf4-f6d248d469b0 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:13.797747 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e71a77-304e-43e8-bbf4-f6d248d469b0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:13.799234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e71a77-304e-43e8-bbf4-f6d248d469b0 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:13.805985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2b4c68-06ff-43fb-8056-8f6ab2ae99f7 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:13.805985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2b4c68-06ff-43fb-8056-8f6ab2ae99f7 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:13.805985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2b4c68-06ff-43fb-8056-8f6ab2ae99f7 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:13.806249 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 366/737] 10.222.0.16 () {56 vars in 1122 bytes} [Sun Oct 30 00:44:13 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:13.809938 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 372/738] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:13 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:13.811973 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2b4c68-06ff-43fb-8056-8f6ab2ae99f7 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:13.812537 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91582f2-9000-47fe-98cc-a31322824606 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:13.812818 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91582f2-9000-47fe-98cc-a31322824606 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:13.813087 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91582f2-9000-47fe-98cc-a31322824606 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:13.831574 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-b91582f2-9000-47fe-98cc-a31322824606 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:13.832862 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b91582f2-9000-47fe-98cc-a31322824606 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:13.907799 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 367/739] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:13 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:13.948934 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2b4c68-06ff-43fb-8056-8f6ab2ae99f7 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:13.983191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024e6993-1366-4e21-95ca-19916da64ebc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:13.983864 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024e6993-1366-4e21-95ca-19916da64ebc None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:13.984141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024e6993-1366-4e21-95ca-19916da64ebc None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:13.984407 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024e6993-1366-4e21-95ca-19916da64ebc None admin] PATH_INFO: `/v3/endpoints` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:13.985161 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024e6993-1366-4e21-95ca-19916da64ebc None admin] RBAC: Adding query filter params (interface=admin, service_id=3e3d7882e3aa4ef99f5b604c5cc12031, region_id=RegionOne) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:13.985469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024e6993-1366-4e21-95ca-19916da64ebc None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:13.986927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024e6993-1366-4e21-95ca-19916da64ebc None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:13.992613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 373/740] 10.222.0.16 () {56 vars in 1249 bytes} [Sun Oct 30 00:44:13 2022] GET /identity/v3/endpoints?service_id=3e3d7882e3aa4ef99f5b604c5cc12031&interface=admin®ion_id=RegionOne => generated 193 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:44:13.998458 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91582f2-9000-47fe-98cc-a31322824606 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:14.105916 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91582f2-9000-47fe-98cc-a31322824606 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:14.107447 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:14.141647 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2dea44-fd4d-490c-90a8-981ac393537b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:14.142320 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2dea44-fd4d-490c-90a8-981ac393537b None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:14.142592 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2dea44-fd4d-490c-90a8-981ac393537b None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:14.142859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2dea44-fd4d-490c-90a8-981ac393537b None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:14.143597 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2dea44-fd4d-490c-90a8-981ac393537b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:14.145063 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2dea44-fd4d-490c-90a8-981ac393537b None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:14.152139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 368/741] 10.222.0.16 () {56 vars in 1087 bytes} [Sun Oct 30 00:44:13 2022] GET /identity/v3/services => generated 1727 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:44:14.204916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024e6993-1366-4e21-95ca-19916da64ebc None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:14.249812 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:14.284170 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62587b2b-d3a4-4a6a-bfbf-0d86271e3beb None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:14.284500 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62587b2b-d3a4-4a6a-bfbf-0d86271e3beb None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:14.284706 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62587b2b-d3a4-4a6a-bfbf-0d86271e3beb None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:14.286334 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 369/742] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:14.291866 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e992935e-bd5b-4a4a-b4b9-22080e4b9e03 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:14.292146 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e992935e-bd5b-4a4a-b4b9-22080e4b9e03 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:14.292413 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e992935e-bd5b-4a4a-b4b9-22080e4b9e03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:14.311680 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-e992935e-bd5b-4a4a-b4b9-22080e4b9e03 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:14.312910 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-e992935e-bd5b-4a4a-b4b9-22080e4b9e03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:14.321719 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024e6993-1366-4e21-95ca-19916da64ebc None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:14.367296 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a067973-1fe8-4ef3-8fb7-6f137141848a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:14.368302 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a067973-1fe8-4ef3-8fb7-6f137141848a None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:14.368678 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a067973-1fe8-4ef3-8fb7-6f137141848a None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:14.369041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a067973-1fe8-4ef3-8fb7-6f137141848a None admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:14.369777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a067973-1fe8-4ef3-8fb7-6f137141848a None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:14.371206 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a067973-1fe8-4ef3-8fb7-6f137141848a None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:14.375243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: WARNING keystone.server.flask.common [None req-5a067973-1fe8-4ef3-8fb7-6f137141848a None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 30 00:44:14.396457 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 370/743] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:14 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 30 00:44:14.400911 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.sql.core [None req-5a067973-1fe8-4ef3-8fb7-6f137141848a None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 30 00:44:14.400911 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 30 00:44:14.400911 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [parameters: {'id': '020b3b8d29754b4e90652b3c4b0e4ec6', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 30 00:44:14.400911 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81495) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 30 00:44:14.410218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-5a067973-1fe8-4ef3-8fb7-6f137141848a None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 020b3b8d29754b4e90652b3c4b0e4ec6.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 020b3b8d29754b4e90652b3c4b0e4ec6. Oct 30 00:44:14.410218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:14.410218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 30 00:44:14.410218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 30 00:44:14.410218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 30 00:44:14.410218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 30 00:44:14.410218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 30 00:44:14.410218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application result = self._query(query) Oct 30 00:44:14.410218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 30 00:44:14.410218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application conn.query(q) Oct 30 00:44:14.410218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 30 00:44:14.410218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 30 00:44:14.410218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 30 00:44:14.411019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application result.read() Oct 30 00:44:14.411019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 30 00:44:14.411019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 30 00:44:14.411019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 30 00:44:14.411019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 30 00:44:14.411019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 30 00:44:14.411019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 30 00:44:14.411019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 30 00:44:14.411019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 30 00:44:14.411019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 30 00:44:14.411019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:14.411019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 30 00:44:14.411019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:14.411019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:14.411019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 30 00:44:14.411019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 30 00:44:14.411019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 30 00:44:14.411019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 30 00:44:14.412020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 30 00:44:14.412020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application next(self.gen) Oct 30 00:44:14.412020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 30 00:44:14.412020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application yield resource Oct 30 00:44:14.412020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 30 00:44:14.412020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application next(self.gen) Oct 30 00:44:14.412020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 30 00:44:14.412020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self.session.rollback() Oct 30 00:44:14.412020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 30 00:44:14.412020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self.force_reraise() Oct 30 00:44:14.412020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 30 00:44:14.412020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise self.value Oct 30 00:44:14.412020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 30 00:44:14.412020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 30 00:44:14.412020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 30 00:44:14.412020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application session.commit() Oct 30 00:44:14.412020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 30 00:44:14.412903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 30 00:44:14.412903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 30 00:44:14.412903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._prepare_impl() Oct 30 00:44:14.412903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 30 00:44:14.412903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self.session.flush() Oct 30 00:44:14.412903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 30 00:44:14.412903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._flush(objects) Oct 30 00:44:14.412903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 30 00:44:14.412903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 30 00:44:14.412903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 30 00:44:14.412903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application compat.raise_( Oct 30 00:44:14.412903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 30 00:44:14.412903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception Oct 30 00:44:14.412903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 30 00:44:14.412903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application flush_context.execute() Oct 30 00:44:14.412903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 30 00:44:14.412903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rec.execute(self) Oct 30 00:44:14.413831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 30 00:44:14.413831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 30 00:44:14.413831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 30 00:44:14.413831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 30 00:44:14.413831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 30 00:44:14.413831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 30 00:44:14.413831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 30 00:44:14.413831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 30 00:44:14.413831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 30 00:44:14.413831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 30 00:44:14.413831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 30 00:44:14.413831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 30 00:44:14.413831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 30 00:44:14.413831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 30 00:44:14.413831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 30 00:44:14.414629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 30 00:44:14.414629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 30 00:44:14.414629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception Oct 30 00:44:14.414629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 30 00:44:14.414629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 30 00:44:14.414629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 30 00:44:14.414629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 30 00:44:14.414629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 30 00:44:14.414629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application result = self._query(query) Oct 30 00:44:14.414629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 30 00:44:14.414629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application conn.query(q) Oct 30 00:44:14.414629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 30 00:44:14.414629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 30 00:44:14.414629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 30 00:44:14.414629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application result.read() Oct 30 00:44:14.414629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 30 00:44:14.414629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 30 00:44:14.415506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 30 00:44:14.415506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 30 00:44:14.415506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 30 00:44:14.415506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 30 00:44:14.415506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 30 00:44:14.415506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 30 00:44:14.415506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 30 00:44:14.415506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 30 00:44:14.415506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [parameters: {'id': '020b3b8d29754b4e90652b3c4b0e4ec6', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 30 00:44:14.415506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 30 00:44:14.415506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:14.415506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:44:14.415506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:14.415506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:14.415506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 30 00:44:14.418120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 30 00:44:14.418120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 30 00:44:14.418120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 30 00:44:14.418120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Oct 30 00:44:14.418120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:14.418120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:44:14.418120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:14.418120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:14.418120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:14.418120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:14.418120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:14.418120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:14.418120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:14.418120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:14.418120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:14.418120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:14.418120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:14.420294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:14.420294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Oct 30 00:44:14.420294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Oct 30 00:44:14.420294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:14.420294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:14.420294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 30 00:44:14.420294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 30 00:44:14.420294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 020b3b8d29754b4e90652b3c4b0e4ec6. Oct 30 00:44:14.420294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:14.420294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 374/744] 10.222.0.16 () {60 vars in 1140 bytes} [Sun Oct 30 00:44:14 2022] POST /identity/v3/projects => generated 262 bytes in 207 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:14.421534 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e992935e-bd5b-4a4a-b4b9-22080e4b9e03 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:14.523598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:14.531217 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e992935e-bd5b-4a4a-b4b9-22080e4b9e03 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:14.568158 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00c9ed6-30eb-4965-904c-50911bbdb5e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:14.568845 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00c9ed6-30eb-4965-904c-50911bbdb5e3 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:14.569127 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00c9ed6-30eb-4965-904c-50911bbdb5e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:14.569394 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00c9ed6-30eb-4965-904c-50911bbdb5e3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:14.576363 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00c9ed6-30eb-4965-904c-50911bbdb5e3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:14.577804 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00c9ed6-30eb-4965-904c-50911bbdb5e3 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:14.589708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-a00c9ed6-30eb-4965-904c-50911bbdb5e3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:44:14.589708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:14.589708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:14.589708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:14.589708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:14.589708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:14.589708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:14.589708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:14.589708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:14.589708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:14.589708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:14.589708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:14.589708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:44:14.589708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:44:14.589708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:44:14.590623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:44:14.590623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:14.590623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:14.590623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:14.590623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:14.590623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:14.590623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:14.590623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:14.590623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:14.590623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:14.590623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:14.590623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:14.590623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:14.590623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:14.590623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:14.590623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:14.590623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:14.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:44:14.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:44:14.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:44:14.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:44:14.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:44:14.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:44:14.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:44:14.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:14.591569 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 371/745] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:44:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:14.598224 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a067973-1fe8-4ef3-8fb7-6f137141848a None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:14.686550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00c9ed6-30eb-4965-904c-50911bbdb5e3 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:14.696172 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:14.706568 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a067973-1fe8-4ef3-8fb7-6f137141848a None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:14.750313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81418c70-4bd3-4699-8c3a-d868eae0e35a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:14.750928 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81418c70-4bd3-4699-8c3a-d868eae0e35a None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:14.751201 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81418c70-4bd3-4699-8c3a-d868eae0e35a None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:14.751464 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81418c70-4bd3-4699-8c3a-d868eae0e35a None admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:14.752988 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81418c70-4bd3-4699-8c3a-d868eae0e35a None admin] RBAC: Adding query filter params (name=service) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:14.753288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81418c70-4bd3-4699-8c3a-d868eae0e35a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:14.755323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81418c70-4bd3-4699-8c3a-d868eae0e35a None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:14.783297 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 375/746] 10.222.0.16 () {56 vars in 1112 bytes} [Sun Oct 30 00:44:14 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:14.806993 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00c9ed6-30eb-4965-904c-50911bbdb5e3 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:14.847686 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8bbe000-846e-4418-a088-bc75a6bf83ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:14.848784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bbe000-846e-4418-a088-bc75a6bf83ec None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:14.849260 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bbe000-846e-4418-a088-bc75a6bf83ec None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:14.849726 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bbe000-846e-4418-a088-bc75a6bf83ec None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:14.850597 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8bbe000-846e-4418-a088-bc75a6bf83ec None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:14.852277 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8bbe000-846e-4418-a088-bc75a6bf83ec None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:14.871390 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 372/747] 10.222.0.16 () {60 vars in 1141 bytes} [Sun Oct 30 00:44:14 2022] POST /identity/v3/services => generated 242 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:44:15.177772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf43a527-7cda-459a-8bd5-36894c854891 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:15.177950 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf43a527-7cda-459a-8bd5-36894c854891 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:15.178213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf43a527-7cda-459a-8bd5-36894c854891 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:15.179826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 376/748] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:15.185103 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe343ced-0025-4da9-9091-d81dffdce5e6 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:15.185565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe343ced-0025-4da9-9091-d81dffdce5e6 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:15.185641 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe343ced-0025-4da9-9091-d81dffdce5e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:15.204533 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-fe343ced-0025-4da9-9091-d81dffdce5e6 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:15.205755 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-fe343ced-0025-4da9-9091-d81dffdce5e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:15.216913 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:15.284080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 373/749] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:15 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:15.543456 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf43a527-7cda-459a-8bd5-36894c854891 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:15.646989 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf43a527-7cda-459a-8bd5-36894c854891 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:15.682204 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf4f3f2-d27b-488c-a5a4-35571147d8e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:15.682833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf4f3f2-d27b-488c-a5a4-35571147d8e5 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:15.683105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf4f3f2-d27b-488c-a5a4-35571147d8e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:15.683373 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf4f3f2-d27b-488c-a5a4-35571147d8e5 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:15.683975 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf4f3f2-d27b-488c-a5a4-35571147d8e5 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:15.685323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf4f3f2-d27b-488c-a5a4-35571147d8e5 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:15.694675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-aaf4f3f2-d27b-488c-a5a4-35571147d8e5 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 30 00:44:15.694675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:15.694675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:44:15.694675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:44:15.694675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:44:15.694675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:44:15.694675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:44:15.694675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:15.694675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 30 00:44:15.694675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:15.694675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:44:15.694675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:15.694675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:15.694675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:15.694675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:15.694675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:15.694675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:15.695673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:15.695673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:15.695673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:15.695673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:15.695673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:15.695673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:15.695673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:44:15.695673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:44:15.695673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:44:15.695673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:44:15.695673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:15.695673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:15.695673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:15.695673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:15.695673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:15.695673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:15.696558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:15.696558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:15.696558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:15.696558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:15.696558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:15.696558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:15.696558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:15.696558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:15.696558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:15.696558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:15.696558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:44:15.696558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:15.696558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 30 00:44:15.696558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:15.696558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 377/750] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:44:15 2022] GET /identity/v3/services/compute => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:15.705200 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe343ced-0025-4da9-9091-d81dffdce5e6 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:15.813003 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe343ced-0025-4da9-9091-d81dffdce5e6 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:15.849093 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e98beb-fcd9-43ee-9610-06d1640a3c4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:15.850070 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e98beb-fcd9-43ee-9610-06d1640a3c4d None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:15.850564 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e98beb-fcd9-43ee-9610-06d1640a3c4d None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:15.851012 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e98beb-fcd9-43ee-9610-06d1640a3c4d None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:15.851887 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e98beb-fcd9-43ee-9610-06d1640a3c4d None admin] RBAC: Adding query filter params (name=compute) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:15.852377 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e98beb-fcd9-43ee-9610-06d1640a3c4d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:15.854007 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e98beb-fcd9-43ee-9610-06d1640a3c4d None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:15.860134 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 374/751] 10.222.0.16 () {56 vars in 1112 bytes} [Sun Oct 30 00:44:15 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:15.869912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf4f3f2-d27b-488c-a5a4-35571147d8e5 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:15.886647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:15.971573 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf4f3f2-d27b-488c-a5a4-35571147d8e5 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:16.004935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d886f37e-2653-4f5e-a422-0ed9232e2a32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:16.005540 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d886f37e-2653-4f5e-a422-0ed9232e2a32 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:16.005819 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d886f37e-2653-4f5e-a422-0ed9232e2a32 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:16.006092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d886f37e-2653-4f5e-a422-0ed9232e2a32 None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:16.006720 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d886f37e-2653-4f5e-a422-0ed9232e2a32 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:16.007016 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d886f37e-2653-4f5e-a422-0ed9232e2a32 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:16.008556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d886f37e-2653-4f5e-a422-0ed9232e2a32 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:16.013935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 378/752] 10.222.0.16 () {56 vars in 1112 bytes} [Sun Oct 30 00:44:15 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:16.019905 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e98beb-fcd9-43ee-9610-06d1640a3c4d None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:16.029140 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:16.125573 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e98beb-fcd9-43ee-9610-06d1640a3c4d None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:16.160093 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb5f3cc-9559-48ec-9319-ea086933e62b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:16.160697 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb5f3cc-9559-48ec-9319-ea086933e62b None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:16.160974 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb5f3cc-9559-48ec-9319-ea086933e62b None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:16.161247 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb5f3cc-9559-48ec-9319-ea086933e62b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:16.161945 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb5f3cc-9559-48ec-9319-ea086933e62b None admin] RBAC: Adding query filter params (interface=public, service_id=192ae356dc664c5885f1384e9857c906, region_id=RegionOne) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:16.162303 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb5f3cc-9559-48ec-9319-ea086933e62b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:16.163658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb5f3cc-9559-48ec-9319-ea086933e62b None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:16.169194 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 375/753] 10.222.0.16 () {56 vars in 1251 bytes} [Sun Oct 30 00:44:16 2022] GET /identity/v3/endpoints?service_id=192ae356dc664c5885f1384e9857c906&interface=public®ion_id=RegionOne => generated 194 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:16.175027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d886f37e-2653-4f5e-a422-0ed9232e2a32 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:16.267789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:16.288818 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d886f37e-2653-4f5e-a422-0ed9232e2a32 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:16.325680 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6692e40e-0f28-49bf-a565-68af7911f5bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:16.326298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6692e40e-0f28-49bf-a565-68af7911f5bc None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:16.326560 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6692e40e-0f28-49bf-a565-68af7911f5bc None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:16.326827 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6692e40e-0f28-49bf-a565-68af7911f5bc None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:16.327417 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6692e40e-0f28-49bf-a565-68af7911f5bc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:16.328819 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6692e40e-0f28-49bf-a565-68af7911f5bc None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:16.335796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 379/754] 10.222.0.16 () {56 vars in 1087 bytes} [Sun Oct 30 00:44:16 2022] GET /identity/v3/services => generated 1957 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:44:16.407516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f59e664-6ddb-47c8-82fd-00e510268086 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:16.408163 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f59e664-6ddb-47c8-82fd-00e510268086 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:16.408163 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f59e664-6ddb-47c8-82fd-00e510268086 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:16.409819 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 376/755] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:16.415554 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fd0f05-b777-4faf-b5fb-f2f00c15ce6c None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:16.415896 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fd0f05-b777-4faf-b5fb-f2f00c15ce6c None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:16.416139 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fd0f05-b777-4faf-b5fb-f2f00c15ce6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:16.436873 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-11fd0f05-b777-4faf-b5fb-f2f00c15ce6c None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:16.438845 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-11fd0f05-b777-4faf-b5fb-f2f00c15ce6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:16.510570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 380/756] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:16 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:16.776339 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f59e664-6ddb-47c8-82fd-00e510268086 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:16.847881 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:16.881838 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f59e664-6ddb-47c8-82fd-00e510268086 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:16.920403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b355346-e88c-4c2b-b96f-2bf22101a707 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:16.921032 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b355346-e88c-4c2b-b96f-2bf22101a707 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:16.921305 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b355346-e88c-4c2b-b96f-2bf22101a707 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:16.921571 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b355346-e88c-4c2b-b96f-2bf22101a707 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:16.922146 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b355346-e88c-4c2b-b96f-2bf22101a707 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:16.923552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b355346-e88c-4c2b-b96f-2bf22101a707 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:16.932797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-5b355346-e88c-4c2b-b96f-2bf22101a707 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 30 00:44:16.932797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:16.932797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:44:16.932797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:44:16.932797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:44:16.932797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:44:16.932797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:44:16.932797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:16.932797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 30 00:44:16.932797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:16.932797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:44:16.932797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:16.932797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:16.932797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:16.932797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:16.932797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:16.932797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:16.933789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:16.933789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:16.933789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:16.933789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:16.933789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:16.933789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:16.933789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:44:16.933789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:44:16.933789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:44:16.933789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:44:16.933789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:16.933789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:16.933789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:16.933789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:16.933789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:16.933789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:16.934632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:16.934632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:16.934632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:16.934632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:16.934632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:16.934632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:16.934632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:16.934632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:16.934632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:16.934632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:16.934632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:44:16.934632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:16.934632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 30 00:44:16.934632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:16.934632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 377/757] 10.222.0.16 () {56 vars in 1126 bytes} [Sun Oct 30 00:44:16 2022] GET /identity/v3/services/object-store => generated 93 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:16.940564 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fd0f05-b777-4faf-b5fb-f2f00c15ce6c None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:17.030220 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:17.048658 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fd0f05-b777-4faf-b5fb-f2f00c15ce6c None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:17.085001 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb0d6c0-c31c-41df-a5c2-9862e4cabcee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:17.086053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb0d6c0-c31c-41df-a5c2-9862e4cabcee None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:17.086557 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb0d6c0-c31c-41df-a5c2-9862e4cabcee None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:17.087019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb0d6c0-c31c-41df-a5c2-9862e4cabcee None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:17.087862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb0d6c0-c31c-41df-a5c2-9862e4cabcee None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:17.088360 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb0d6c0-c31c-41df-a5c2-9862e4cabcee None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:17.089905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb0d6c0-c31c-41df-a5c2-9862e4cabcee None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:17.095919 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 381/758] 10.222.0.16 () {56 vars in 1122 bytes} [Sun Oct 30 00:44:16 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:17.105303 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b355346-e88c-4c2b-b96f-2bf22101a707 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:17.149460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0753db1-f25a-48a9-bce3-baab1c37be4a None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:17.149795 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0753db1-f25a-48a9-bce3-baab1c37be4a None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:17.150007 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0753db1-f25a-48a9-bce3-baab1c37be4a None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:17.151383 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 382/759] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:44:17.156850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf53820-4ce5-46b9-9bae-d5a0ff5873b4 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:17.157292 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf53820-4ce5-46b9-9bae-d5a0ff5873b4 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:17.157632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf53820-4ce5-46b9-9bae-d5a0ff5873b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:17.177582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-ddf53820-4ce5-46b9-9bae-d5a0ff5873b4 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:17.179055 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ddf53820-4ce5-46b9-9bae-d5a0ff5873b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:17.219951 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b355346-e88c-4c2b-b96f-2bf22101a707 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:17.257823 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b36fea-ce2b-4a25-9dd9-041ff3aceb0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:17.264575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b36fea-ce2b-4a25-9dd9-041ff3aceb0f None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:17.264770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 383/760] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:17 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 30 00:44:17.264882 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b36fea-ce2b-4a25-9dd9-041ff3aceb0f None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:17.264882 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b36fea-ce2b-4a25-9dd9-041ff3aceb0f None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:17.264882 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b36fea-ce2b-4a25-9dd9-041ff3aceb0f None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:17.265067 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617aeb91-abad-4435-98ba-dc8cafab2024 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:17.265135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b36fea-ce2b-4a25-9dd9-041ff3aceb0f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:17.265212 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617aeb91-abad-4435-98ba-dc8cafab2024 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:17.265212 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617aeb91-abad-4435-98ba-dc8cafab2024 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:17.265335 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b36fea-ce2b-4a25-9dd9-041ff3aceb0f None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:17.265402 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 384/761] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:17.267374 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:17.267489 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed571521-a258-41b2-a292-bdbd5482538c None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:17.267818 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed571521-a258-41b2-a292-bdbd5482538c None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:17.268089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed571521-a258-41b2-a292-bdbd5482538c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:17.271322 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 378/762] 10.222.0.16 () {56 vars in 1122 bytes} [Sun Oct 30 00:44:17 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:17.280154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b36fea-ce2b-4a25-9dd9-041ff3aceb0f None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:17.286808 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-ed571521-a258-41b2-a292-bdbd5482538c None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:17.288005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ed571521-a258-41b2-a292-bdbd5482538c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:17.360401 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:17.371772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 385/763] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:17 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 30 00:44:17.400951 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b36fea-ce2b-4a25-9dd9-041ff3aceb0f None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:17.436695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ba87df-39c0-4610-bf30-6ad667276aa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:17.438167 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ba87df-39c0-4610-bf30-6ad667276aa9 None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:17.438636 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ba87df-39c0-4610-bf30-6ad667276aa9 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:17.439133 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ba87df-39c0-4610-bf30-6ad667276aa9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:17.440069 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ba87df-39c0-4610-bf30-6ad667276aa9 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:17.441668 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ba87df-39c0-4610-bf30-6ad667276aa9 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:17.461761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 379/764] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:44:17 2022] POST /identity/v3/endpoints => generated 325 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:44:17.567227 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed571521-a258-41b2-a292-bdbd5482538c None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:17.640846 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ba87df-39c0-4610-bf30-6ad667276aa9 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:17.684452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed571521-a258-41b2-a292-bdbd5482538c None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:17.716101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:17.726699 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89d28a3-9a2b-48ff-8833-ac1b5f37471c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:17.727442 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89d28a3-9a2b-48ff-8833-ac1b5f37471c None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:17.727834 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89d28a3-9a2b-48ff-8833-ac1b5f37471c None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:17.728197 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89d28a3-9a2b-48ff-8833-ac1b5f37471c None admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:17.728929 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f89d28a3-9a2b-48ff-8833-ac1b5f37471c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:17.730426 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f89d28a3-9a2b-48ff-8833-ac1b5f37471c None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:17.745805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.sql.core [None req-f89d28a3-9a2b-48ff-8833-ac1b5f37471c None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 30 00:44:17.745805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 30 00:44:17.745805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [parameters: {'user_id': '0e4b57b952a247518b8df70e60438b3a', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 30 00:44:17.745805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81495) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 30 00:44:17.754300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-f89d28a3-9a2b-48ff-8833-ac1b5f37471c None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 30 00:44:17.754300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:17.754300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 30 00:44:17.754300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 30 00:44:17.754300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 30 00:44:17.754300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 30 00:44:17.754300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 30 00:44:17.754300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application result = self._query(query) Oct 30 00:44:17.754300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 30 00:44:17.754300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application conn.query(q) Oct 30 00:44:17.754300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 30 00:44:17.754300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 30 00:44:17.754300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 30 00:44:17.754300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application result.read() Oct 30 00:44:17.754300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 30 00:44:17.755259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 30 00:44:17.755259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 30 00:44:17.755259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 30 00:44:17.755259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 30 00:44:17.755259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 30 00:44:17.755259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 30 00:44:17.755259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 30 00:44:17.755259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 30 00:44:17.755259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:17.755259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 30 00:44:17.755259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:17.755259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:17.755259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 30 00:44:17.755259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 30 00:44:17.755259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Oct 30 00:44:17.755259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Oct 30 00:44:17.755259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 30 00:44:17.755259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application next(self.gen) Oct 30 00:44:17.756457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 30 00:44:17.756457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application yield resource Oct 30 00:44:17.756457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 30 00:44:17.756457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application next(self.gen) Oct 30 00:44:17.756457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 30 00:44:17.756457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self.session.rollback() Oct 30 00:44:17.756457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 30 00:44:17.756457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self.force_reraise() Oct 30 00:44:17.756457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 30 00:44:17.756457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise self.value Oct 30 00:44:17.756457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 30 00:44:17.756457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 30 00:44:17.756457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 30 00:44:17.756457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application session.commit() Oct 30 00:44:17.756457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 30 00:44:17.756457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 30 00:44:17.756457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 30 00:44:17.757547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._prepare_impl() Oct 30 00:44:17.757547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 30 00:44:17.757547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self.session.flush() Oct 30 00:44:17.757547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 30 00:44:17.757547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._flush(objects) Oct 30 00:44:17.757547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 30 00:44:17.757547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 30 00:44:17.757547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 30 00:44:17.757547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application compat.raise_( Oct 30 00:44:17.757547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 30 00:44:17.757547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception Oct 30 00:44:17.757547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 30 00:44:17.757547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application flush_context.execute() Oct 30 00:44:17.757547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 30 00:44:17.757547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rec.execute(self) Oct 30 00:44:17.757547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 30 00:44:17.757547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 30 00:44:17.758620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 30 00:44:17.758620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 30 00:44:17.758620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Oct 30 00:44:17.758620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application result = connection._execute_20( Oct 30 00:44:17.758620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 30 00:44:17.758620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 30 00:44:17.758620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 30 00:44:17.758620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 30 00:44:17.758620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 30 00:44:17.758620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 30 00:44:17.758620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 30 00:44:17.758620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 30 00:44:17.758620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 30 00:44:17.758620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 30 00:44:17.758620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 30 00:44:17.758620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception Oct 30 00:44:17.759647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 30 00:44:17.759647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 30 00:44:17.759647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 30 00:44:17.759647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 30 00:44:17.759647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 30 00:44:17.759647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application result = self._query(query) Oct 30 00:44:17.759647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 30 00:44:17.759647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application conn.query(q) Oct 30 00:44:17.759647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 30 00:44:17.759647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 30 00:44:17.759647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 30 00:44:17.759647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application result.read() Oct 30 00:44:17.759647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 30 00:44:17.759647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 30 00:44:17.759647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 30 00:44:17.759647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 30 00:44:17.760683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 30 00:44:17.760683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 30 00:44:17.760683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 30 00:44:17.760683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 30 00:44:17.760683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 30 00:44:17.760683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 30 00:44:17.760683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [parameters: {'user_id': '0e4b57b952a247518b8df70e60438b3a', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 30 00:44:17.760683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 30 00:44:17.760683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:17.760683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:44:17.760683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:17.760683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:17.760683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:17.760683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:17.760683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:17.761650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:17.761650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:17.761650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:17.761650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:17.761650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:17.761650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:17.761650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:17.761650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Oct 30 00:44:17.761650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Oct 30 00:44:17.761650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:17.761650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:17.761650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:44:17.761650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:44:17.761650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:44:17.761650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:44:17.761650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Oct 30 00:44:17.761650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Oct 30 00:44:17.762779 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Oct 30 00:44:17.762779 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Oct 30 00:44:17.762779 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 30 00:44:17.762779 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 30 00:44:17.762779 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 30 00:44:17.762779 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:17.762779 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 386/765] 10.222.0.16 () {60 vars in 1131 bytes} [Sun Oct 30 00:44:17 2022] POST /identity/v3/users => generated 160 bytes in 190 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:17.768215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ba87df-39c0-4610-bf30-6ad667276aa9 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:17.773684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89d28a3-9a2b-48ff-8833-ac1b5f37471c None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:17.806978 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3e4b4d-d237-4754-a48f-8c6d9b3ed2ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:17.807809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3e4b4d-d237-4754-a48f-8c6d9b3ed2ca None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:17.808136 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3e4b4d-d237-4754-a48f-8c6d9b3ed2ca None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:17.808409 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3e4b4d-d237-4754-a48f-8c6d9b3ed2ca None admin] PATH_INFO: `/v3/services/image` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:17.808995 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3e4b4d-d237-4754-a48f-8c6d9b3ed2ca None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:17.810898 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3e4b4d-d237-4754-a48f-8c6d9b3ed2ca None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:17.822092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-2f3e4b4d-d237-4754-a48f-8c6d9b3ed2ca None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 30 00:44:17.822092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:17.822092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:44:17.822092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:44:17.822092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:44:17.822092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:44:17.822092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:44:17.822092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:17.822092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 30 00:44:17.822092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:17.822092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:44:17.822092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:17.822092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:17.822092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:17.822092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:17.822092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:17.822092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:17.823117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:17.823117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:17.823117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:17.823117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:17.823117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:17.823117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:17.823117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:44:17.823117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:44:17.823117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:44:17.823117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:44:17.823117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:17.823117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:17.823117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:17.823117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:17.823117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:17.823117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:17.824636 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:17.824636 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:17.824636 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:17.824636 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:17.824636 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:17.824636 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:17.824636 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:17.824636 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:17.824636 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:17.824636 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:17.824636 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:44:17.824636 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:17.824636 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 30 00:44:17.824636 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:17.824636 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 380/766] 10.222.0.16 () {56 vars in 1105 bytes} [Sun Oct 30 00:44:17 2022] GET /identity/v3/services/image => generated 86 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:44:17.828854 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3e4b4d-d237-4754-a48f-8c6d9b3ed2ca None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:17.848282 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:17.896016 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89d28a3-9a2b-48ff-8833-ac1b5f37471c None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:17.931044 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3e4b4d-d237-4754-a48f-8c6d9b3ed2ca None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:17.932782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0477bd-6c4a-4c1a-87be-944fe6337015 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:17.933421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0477bd-6c4a-4c1a-87be-944fe6337015 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:17.933691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0477bd-6c4a-4c1a-87be-944fe6337015 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:17.933955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0477bd-6c4a-4c1a-87be-944fe6337015 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:17.942182 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0477bd-6c4a-4c1a-87be-944fe6337015 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:17.943895 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0477bd-6c4a-4c1a-87be-944fe6337015 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:17.956482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-da0477bd-6c4a-4c1a-87be-944fe6337015 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 30 00:44:17.956482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:17.956482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:17.956482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:17.956482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:17.956482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:17.956482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:17.956482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:17.956482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:17.956482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:17.956482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:17.956482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:17.956482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:44:17.956482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:44:17.956482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:44:17.956482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:44:17.957458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:17.957458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:17.957458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:44:17.957458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:44:17.957458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:44:17.957458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:44:17.957458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:17.957458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:17.957458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:17.957458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:17.957458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:17.957458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:17.957458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:17.957458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:17.957458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:17.957458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:17.957458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:17.957458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:17.958400 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:17.958400 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:17.958400 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:44:17.958400 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:44:17.958400 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:44:17.958400 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:44:17.958400 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:44:17.958400 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:44:17.958400 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 30 00:44:17.958400 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:17.958400 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 387/767] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:44:17 2022] GET /identity/v3/users/cinder => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:17.960364 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:17.963937 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0477bd-6c4a-4c1a-87be-944fe6337015 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:17.988412 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a8784a-c095-47d9-ae59-def6ca4339fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:17.989030 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a8784a-c095-47d9-ae59-def6ca4339fa None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:17.989302 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a8784a-c095-47d9-ae59-def6ca4339fa None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:17.989568 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a8784a-c095-47d9-ae59-def6ca4339fa None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:17.990192 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a8784a-c095-47d9-ae59-def6ca4339fa None admin] RBAC: Adding query filter params (name=image) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:17.990487 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a8784a-c095-47d9-ae59-def6ca4339fa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:17.991900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a8784a-c095-47d9-ae59-def6ca4339fa None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:18.001875 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 381/768] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:44:17 2022] GET /identity/v3/services?name=image => generated 122 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:44:18.002901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a8784a-c095-47d9-ae59-def6ca4339fa None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:18.084170 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0477bd-6c4a-4c1a-87be-944fe6337015 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:18.107053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:18.128619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c544d42b-f317-4c4c-abcc-8767ca9ca748 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:18.129104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a8784a-c095-47d9-ae59-def6ca4339fa None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:18.129845 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c544d42b-f317-4c4c-abcc-8767ca9ca748 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:18.130315 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c544d42b-f317-4c4c-abcc-8767ca9ca748 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:18.130972 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c544d42b-f317-4c4c-abcc-8767ca9ca748 None admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:18.132797 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c544d42b-f317-4c4c-abcc-8767ca9ca748 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:18.133299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c544d42b-f317-4c4c-abcc-8767ca9ca748 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:18.134900 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c544d42b-f317-4c4c-abcc-8767ca9ca748 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:18.164895 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f061eac5-c906-4122-a88b-1c4ba043cfb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:18.165107 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f061eac5-c906-4122-a88b-1c4ba043cfb5 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:18.165463 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f061eac5-c906-4122-a88b-1c4ba043cfb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:18.165798 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f061eac5-c906-4122-a88b-1c4ba043cfb5 None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:18.166484 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f061eac5-c906-4122-a88b-1c4ba043cfb5 None admin] RBAC: Adding query filter params (type=image) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:18.166895 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f061eac5-c906-4122-a88b-1c4ba043cfb5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:18.168334 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f061eac5-c906-4122-a88b-1c4ba043cfb5 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:18.178472 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 382/769] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:44:18 2022] GET /identity/v3/services?type=image => generated 350 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:18.180059 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f061eac5-c906-4122-a88b-1c4ba043cfb5 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:18.187894 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 388/770] 10.222.0.16 () {56 vars in 1101 bytes} [Sun Oct 30 00:44:17 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:18.239678 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:18.301760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f061eac5-c906-4122-a88b-1c4ba043cfb5 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:18.340219 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99dc95c9-5802-4555-a3a0-1cad246ece45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:18.341207 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dc95c9-5802-4555-a3a0-1cad246ece45 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:18.341710 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dc95c9-5802-4555-a3a0-1cad246ece45 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:18.342184 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dc95c9-5802-4555-a3a0-1cad246ece45 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:18.343100 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99dc95c9-5802-4555-a3a0-1cad246ece45 None admin] RBAC: Adding query filter params (interface=public, service_id=d285b6e84c97405ead16bf3dc9a24470, region_id=RegionOne) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:18.343602 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99dc95c9-5802-4555-a3a0-1cad246ece45 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:18.345219 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99dc95c9-5802-4555-a3a0-1cad246ece45 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:18.352462 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 383/771] 10.222.0.16 () {56 vars in 1251 bytes} [Sun Oct 30 00:44:18 2022] GET /identity/v3/endpoints?service_id=d285b6e84c97405ead16bf3dc9a24470&interface=public®ion_id=RegionOne => generated 194 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:18.360849 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c544d42b-f317-4c4c-abcc-8767ca9ca748 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:18.468210 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c544d42b-f317-4c4c-abcc-8767ca9ca748 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:18.509701 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3696393b-570a-485f-ab45-86aef83e9543 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:18.510315 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3696393b-570a-485f-ab45-86aef83e9543 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:18.510588 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3696393b-570a-485f-ab45-86aef83e9543 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:18.510854 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3696393b-570a-485f-ab45-86aef83e9543 None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:18.511510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3696393b-570a-485f-ab45-86aef83e9543 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:18.512911 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3696393b-570a-485f-ab45-86aef83e9543 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:18.520272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 389/772] 10.222.0.16 () {56 vars in 1087 bytes} [Sun Oct 30 00:44:18 2022] GET /identity/v3/services => generated 1957 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:44:18.592767 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551aaff1-0255-423d-8c46-8aa8fb2a4a41 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:18.593282 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551aaff1-0255-423d-8c46-8aa8fb2a4a41 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:18.593282 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551aaff1-0255-423d-8c46-8aa8fb2a4a41 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:18.594781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 384/773] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:18.600963 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9bba43-f1b5-4d84-a51c-03e4715508e9 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:18.601327 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9bba43-f1b5-4d84-a51c-03e4715508e9 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:18.601518 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9bba43-f1b5-4d84-a51c-03e4715508e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:18.620913 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-8e9bba43-f1b5-4d84-a51c-03e4715508e9 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:18.622463 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-8e9bba43-f1b5-4d84-a51c-03e4715508e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:18.661522 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:18.699196 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 390/774] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:18 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:18.972253 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551aaff1-0255-423d-8c46-8aa8fb2a4a41 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:19.082394 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551aaff1-0255-423d-8c46-8aa8fb2a4a41 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:19.096830 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:19.119218 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1753e5d1-8c31-4020-aeae-6129b855f9d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:19.119872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1753e5d1-8c31-4020-aeae-6129b855f9d6 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:19.120149 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1753e5d1-8c31-4020-aeae-6129b855f9d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:19.120413 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1753e5d1-8c31-4020-aeae-6129b855f9d6 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:19.120988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1753e5d1-8c31-4020-aeae-6129b855f9d6 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:19.122365 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1753e5d1-8c31-4020-aeae-6129b855f9d6 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:19.131901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-1753e5d1-8c31-4020-aeae-6129b855f9d6 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 30 00:44:19.131901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:19.131901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:44:19.131901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:44:19.131901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:44:19.131901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:44:19.131901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:44:19.131901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:19.131901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 30 00:44:19.131901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:19.131901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:44:19.131901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:19.131901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:19.131901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:19.131901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:19.131901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:19.131901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:19.132859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:19.132859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:19.132859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:19.132859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:19.132859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:19.132859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:19.132859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:44:19.132859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:44:19.132859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:44:19.132859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:44:19.132859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:19.132859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:19.132859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:19.132859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:19.132859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:19.132859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:19.133715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:19.133715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:19.133715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:19.133715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:19.133715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:19.133715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:19.133715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:19.133715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:19.133715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:19.133715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:19.133715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:44:19.133715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:19.133715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 30 00:44:19.133715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:19.135211 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 385/775] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:44:18 2022] GET /identity/v3/services/compute => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:19.139091 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9bba43-f1b5-4d84-a51c-03e4715508e9 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:19.246978 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9bba43-f1b5-4d84-a51c-03e4715508e9 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:19.282586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2659bf2a-16ef-428d-8510-91c3bfa455b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:19.283254 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2659bf2a-16ef-428d-8510-91c3bfa455b7 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:19.283662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2659bf2a-16ef-428d-8510-91c3bfa455b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:19.283966 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2659bf2a-16ef-428d-8510-91c3bfa455b7 None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:19.284587 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2659bf2a-16ef-428d-8510-91c3bfa455b7 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:19.284884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2659bf2a-16ef-428d-8510-91c3bfa455b7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:19.286246 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2659bf2a-16ef-428d-8510-91c3bfa455b7 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:19.295935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 391/776] 10.222.0.16 () {56 vars in 1112 bytes} [Sun Oct 30 00:44:19 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:19.297529 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1753e5d1-8c31-4020-aeae-6129b855f9d6 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:19.411365 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1753e5d1-8c31-4020-aeae-6129b855f9d6 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:19.447553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896e954c-8fca-4f3b-b647-633019b2a920 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:19.448205 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896e954c-8fca-4f3b-b647-633019b2a920 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:19.448477 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896e954c-8fca-4f3b-b647-633019b2a920 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:19.448746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896e954c-8fca-4f3b-b647-633019b2a920 None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:19.449375 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896e954c-8fca-4f3b-b647-633019b2a920 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:19.449673 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896e954c-8fca-4f3b-b647-633019b2a920 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:19.451034 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896e954c-8fca-4f3b-b647-633019b2a920 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:19.456331 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 386/777] 10.222.0.16 () {56 vars in 1112 bytes} [Sun Oct 30 00:44:19 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:19.462127 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2659bf2a-16ef-428d-8510-91c3bfa455b7 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:19.500631 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:19.570092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2659bf2a-16ef-428d-8510-91c3bfa455b7 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:19.604276 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76461787-e478-47c7-9b1a-8f013af01297 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:19.604895 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76461787-e478-47c7-9b1a-8f013af01297 None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:19.605165 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76461787-e478-47c7-9b1a-8f013af01297 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:19.605428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76461787-e478-47c7-9b1a-8f013af01297 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:19.606099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76461787-e478-47c7-9b1a-8f013af01297 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:19.607517 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76461787-e478-47c7-9b1a-8f013af01297 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:19.639747 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 392/778] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:44:19 2022] POST /identity/v3/endpoints => generated 334 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:44:19.727857 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d024eb3-e82c-41c0-b7ad-d6d0d48a9e01 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:19.728384 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d024eb3-e82c-41c0-b7ad-d6d0d48a9e01 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:19.728498 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d024eb3-e82c-41c0-b7ad-d6d0d48a9e01 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:19.729856 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 387/779] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:19.735683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeac588-da09-4e33-83b5-7e675cc105d3 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:19.736077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeac588-da09-4e33-83b5-7e675cc105d3 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:19.736346 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeac588-da09-4e33-83b5-7e675cc105d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:19.756948 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-edeac588-da09-4e33-83b5-7e675cc105d3 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:19.758071 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-edeac588-da09-4e33-83b5-7e675cc105d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:19.839066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 393/780] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:19 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:20.107081 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d024eb3-e82c-41c0-b7ad-d6d0d48a9e01 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:20.156284 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:20.222480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d024eb3-e82c-41c0-b7ad-d6d0d48a9e01 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:20.258265 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1159dd0e-95dd-4974-bb26-09428b305ace None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:20.258882 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1159dd0e-95dd-4974-bb26-09428b305ace None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:20.259153 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1159dd0e-95dd-4974-bb26-09428b305ace None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:20.259413 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1159dd0e-95dd-4974-bb26-09428b305ace None admin] PATH_INFO: `/v3/domains/default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:20.260524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1159dd0e-95dd-4974-bb26-09428b305ace None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:20.261938 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1159dd0e-95dd-4974-bb26-09428b305ace None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:20.263671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 388/781] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:44:20 2022] GET /identity/v3/domains/default => generated 199 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:20.269484 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edeac588-da09-4e33-83b5-7e675cc105d3 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:20.287413 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:20.377559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edeac588-da09-4e33-83b5-7e675cc105d3 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:20.412022 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65adfbb-495f-42c5-955f-e4f309b449de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:20.412627 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65adfbb-495f-42c5-955f-e4f309b449de None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:20.412897 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65adfbb-495f-42c5-955f-e4f309b449de None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:20.413158 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65adfbb-495f-42c5-955f-e4f309b449de None admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:20.413805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65adfbb-495f-42c5-955f-e4f309b449de None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:20.415221 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65adfbb-495f-42c5-955f-e4f309b449de None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:20.447760 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 394/782] 10.222.0.16 () {60 vars in 1141 bytes} [Sun Oct 30 00:44:20 2022] POST /identity/v3/projects => generated 311 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:44:20.656119 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c632576-119c-4edd-9913-98a3fb7c2bd6 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:20.656306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c632576-119c-4edd-9913-98a3fb7c2bd6 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:20.656566 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c632576-119c-4edd-9913-98a3fb7c2bd6 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:20.658096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 389/783] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:20.663677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189688d2-0ab2-456b-b385-1fc2c23397ae None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:20.664013 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189688d2-0ab2-456b-b385-1fc2c23397ae None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:20.664281 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189688d2-0ab2-456b-b385-1fc2c23397ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:20.683284 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-189688d2-0ab2-456b-b385-1fc2c23397ae None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:20.684493 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-189688d2-0ab2-456b-b385-1fc2c23397ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:20.752090 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:20.758594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 395/784] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:20 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:21.020942 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c632576-119c-4edd-9913-98a3fb7c2bd6 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:21.125404 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c632576-119c-4edd-9913-98a3fb7c2bd6 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:21.136212 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:21.161335 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9acd4b-ae9a-4e52-b1cf-199b3132833e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:21.161963 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9acd4b-ae9a-4e52-b1cf-199b3132833e None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:21.162237 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9acd4b-ae9a-4e52-b1cf-199b3132833e None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:21.162503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9acd4b-ae9a-4e52-b1cf-199b3132833e None admin] PATH_INFO: `/v3/services/image` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:21.163097 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9acd4b-ae9a-4e52-b1cf-199b3132833e None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:21.164485 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9acd4b-ae9a-4e52-b1cf-199b3132833e None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:21.173336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-4a9acd4b-ae9a-4e52-b1cf-199b3132833e None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 30 00:44:21.173336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:21.173336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:44:21.173336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:44:21.173336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:44:21.173336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:44:21.173336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:44:21.173336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:21.173336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 30 00:44:21.173336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:21.173336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:44:21.173336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:21.173336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:21.173336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:21.173336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:21.173336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:21.173336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:21.174439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:21.174439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:21.174439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:21.174439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:21.174439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:21.174439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:21.174439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:44:21.174439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:44:21.174439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:44:21.174439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:44:21.174439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:21.174439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:21.174439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:21.174439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:21.174439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:21.174439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:21.175272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:21.175272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:21.175272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:21.175272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:21.175272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:21.175272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:21.175272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:21.175272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:21.175272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:21.175272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:21.175272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:44:21.175272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:44:21.175272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 30 00:44:21.175272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:21.175272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 390/785] 10.222.0.16 () {56 vars in 1105 bytes} [Sun Oct 30 00:44:21 2022] GET /identity/v3/services/image => generated 86 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:21.181734 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189688d2-0ab2-456b-b385-1fc2c23397ae None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:21.281324 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189688d2-0ab2-456b-b385-1fc2c23397ae None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:21.315198 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2042838-df48-4cc1-ad1f-cd812e7ff14d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:21.315828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2042838-df48-4cc1-ad1f-cd812e7ff14d None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:21.316100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2042838-df48-4cc1-ad1f-cd812e7ff14d None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:21.316361 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2042838-df48-4cc1-ad1f-cd812e7ff14d None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:21.316977 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2042838-df48-4cc1-ad1f-cd812e7ff14d None admin] RBAC: Adding query filter params (name=image) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:21.317272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2042838-df48-4cc1-ad1f-cd812e7ff14d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:21.318617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2042838-df48-4cc1-ad1f-cd812e7ff14d None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:21.328326 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 396/786] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:44:21 2022] GET /identity/v3/services?name=image => generated 122 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:21.329564 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9acd4b-ae9a-4e52-b1cf-199b3132833e None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:21.442115 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9acd4b-ae9a-4e52-b1cf-199b3132833e None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:21.476791 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6e9874-2420-42b0-854c-cdf98bdc9d0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:21.477392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6e9874-2420-42b0-854c-cdf98bdc9d0b None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:21.477665 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6e9874-2420-42b0-854c-cdf98bdc9d0b None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:21.477933 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6e9874-2420-42b0-854c-cdf98bdc9d0b None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:21.478551 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f6e9874-2420-42b0-854c-cdf98bdc9d0b None admin] RBAC: Adding query filter params (type=image) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:21.478851 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f6e9874-2420-42b0-854c-cdf98bdc9d0b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:21.480246 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f6e9874-2420-42b0-854c-cdf98bdc9d0b None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:21.485815 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 391/787] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:44:21 2022] GET /identity/v3/services?type=image => generated 350 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:21.491523 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2042838-df48-4cc1-ad1f-cd812e7ff14d None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:21.553480 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:21.593401 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2042838-df48-4cc1-ad1f-cd812e7ff14d None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:21.628764 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4faa221-d07f-4b21-907d-b1374e85d2fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:21.629376 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4faa221-d07f-4b21-907d-b1374e85d2fe None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:21.629684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4faa221-d07f-4b21-907d-b1374e85d2fe None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:21.629947 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4faa221-d07f-4b21-907d-b1374e85d2fe None admin] PATH_INFO: `/v3/endpoints` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:21.630647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4faa221-d07f-4b21-907d-b1374e85d2fe None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:21.632356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4faa221-d07f-4b21-907d-b1374e85d2fe None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:21.653354 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 397/788] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:44:21 2022] POST /identity/v3/endpoints => generated 327 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:44:21.894188 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75787d3-8f5c-42fd-9f10-f69a655be152 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:21.894396 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75787d3-8f5c-42fd-9f10-f69a655be152 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:21.894667 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75787d3-8f5c-42fd-9f10-f69a655be152 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:21.896177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 392/789] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:21.901464 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48ddc2f-0700-4542-b6e8-8d4db0aa92d5 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:21.901740 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48ddc2f-0700-4542-b6e8-8d4db0aa92d5 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:21.902009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48ddc2f-0700-4542-b6e8-8d4db0aa92d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:21.920939 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-e48ddc2f-0700-4542-b6e8-8d4db0aa92d5 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:21.922113 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e48ddc2f-0700-4542-b6e8-8d4db0aa92d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:22.011106 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 398/790] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:22.301468 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75787d3-8f5c-42fd-9f10-f69a655be152 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:22.351930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:22.426346 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75787d3-8f5c-42fd-9f10-f69a655be152 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:22.463115 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12102a02-c545-40dd-b9f0-231b1e2f15d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:22.463747 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12102a02-c545-40dd-b9f0-231b1e2f15d9 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:22.464031 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12102a02-c545-40dd-b9f0-231b1e2f15d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:22.464297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12102a02-c545-40dd-b9f0-231b1e2f15d9 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:22.475888 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12102a02-c545-40dd-b9f0-231b1e2f15d9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:44:22.480632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-12102a02-c545-40dd-b9f0-231b1e2f15d9 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 30 00:44:22.480632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:22.480632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:22.480632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:22.480632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:22.480632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:22.480632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:22.480632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:22.480632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:22.480632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:22.480632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:22.480632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:22.480632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:44:22.480632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:44:22.480632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:44:22.480632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:44:22.481567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:44:22.481567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:44:22.481567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:44:22.481567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:44:22.481567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:22.481567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:22.481567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:22.481567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:22.481567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:22.481567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:22.481567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:22.481567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:22.481567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:22.481567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:22.481567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:22.481567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:22.482392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:22.482392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:22.482392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:22.482392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:22.482392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:44:22.482392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:44:22.482392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:44:22.482392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:44:22.482392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:44:22.482392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:44:22.482392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 30 00:44:22.482392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:22.482392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 393/791] 10.222.0.16 () {56 vars in 1120 bytes} [Sun Oct 30 00:44:22 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:22.492825 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48ddc2f-0700-4542-b6e8-8d4db0aa92d5 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:22.543900 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:22.599480 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48ddc2f-0700-4542-b6e8-8d4db0aa92d5 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:22.629214 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85907f8-5102-4c4b-af29-2905c36cceb7 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:22.629606 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85907f8-5102-4c4b-af29-2905c36cceb7 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:22.629939 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85907f8-5102-4c4b-af29-2905c36cceb7 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:22.631454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 394/792] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:22.637449 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c8a017-adcc-4068-b3dc-732b9f6c1590 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:22.637713 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c8a017-adcc-4068-b3dc-732b9f6c1590 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:22.637996 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c8a017-adcc-4068-b3dc-732b9f6c1590 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:22.641856 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f87e55-3ec6-48d1-812c-eb20356bb966 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:22.642707 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f87e55-3ec6-48d1-812c-eb20356bb966 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:22.643022 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f87e55-3ec6-48d1-812c-eb20356bb966 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:22.643325 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f87e55-3ec6-48d1-812c-eb20356bb966 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:22.644014 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f87e55-3ec6-48d1-812c-eb20356bb966 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:22.644311 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f87e55-3ec6-48d1-812c-eb20356bb966 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:22.645704 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f87e55-3ec6-48d1-812c-eb20356bb966 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:22.659456 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 399/793] 10.222.0.16 () {56 vars in 1115 bytes} [Sun Oct 30 00:44:22 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:22.659646 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-00c8a017-adcc-4068-b3dc-732b9f6c1590 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:22.659646 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-00c8a017-adcc-4068-b3dc-732b9f6c1590 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:22.660511 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f87e55-3ec6-48d1-812c-eb20356bb966 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:22.732079 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 395/794] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:22 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 30 00:44:22.768061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f87e55-3ec6-48d1-812c-eb20356bb966 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:22.775668 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:22.803834 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c89869-1630-4f01-96d0-d1e0554d986b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:22.804459 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c89869-1630-4f01-96d0-d1e0554d986b None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:22.804732 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c89869-1630-4f01-96d0-d1e0554d986b None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:22.804995 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c89869-1630-4f01-96d0-d1e0554d986b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:22.815933 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c89869-1630-4f01-96d0-d1e0554d986b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:22.817357 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c89869-1630-4f01-96d0-d1e0554d986b None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:22.831275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-13c89869-1630-4f01-96d0-d1e0554d986b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:44:22.831275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:22.831275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:22.831275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:22.831275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:22.831275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:22.831275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:22.831275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:22.831275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:22.831275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:22.831275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:22.831275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:22.831275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:44:22.831275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:44:22.831275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:44:22.831275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:44:22.832419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:22.832419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:22.832419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:22.832419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:22.832419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:22.832419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:22.832419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:22.832419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:22.832419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:22.832419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:22.832419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:22.832419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:22.832419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:22.832419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:22.832419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:22.832419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:22.832419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:44:22.833347 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:44:22.833347 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:44:22.833347 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:22.833995 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 400/795] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:44:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:22.838197 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c8a017-adcc-4068-b3dc-732b9f6c1590 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:22.868104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:22.941917 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c8a017-adcc-4068-b3dc-732b9f6c1590 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:22.980817 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406862ed-8b7f-45b7-8299-9e949e091a69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:22.981419 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406862ed-8b7f-45b7-8299-9e949e091a69 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:22.981688 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406862ed-8b7f-45b7-8299-9e949e091a69 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:22.981954 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406862ed-8b7f-45b7-8299-9e949e091a69 None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:22.982588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406862ed-8b7f-45b7-8299-9e949e091a69 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:22.982882 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406862ed-8b7f-45b7-8299-9e949e091a69 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:22.984269 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406862ed-8b7f-45b7-8299-9e949e091a69 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:22.994685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c89869-1630-4f01-96d0-d1e0554d986b None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:22.997450 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 396/796] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:44:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:23.002758 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406862ed-8b7f-45b7-8299-9e949e091a69 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:23.099343 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:23.108506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c89869-1630-4f01-96d0-d1e0554d986b None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:23.113463 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406862ed-8b7f-45b7-8299-9e949e091a69 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:23.142417 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d09ac0-aaa1-494e-a6f0-19e233faa92c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:23.143044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d09ac0-aaa1-494e-a6f0-19e233faa92c None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:23.143311 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d09ac0-aaa1-494e-a6f0-19e233faa92c None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:23.143587 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d09ac0-aaa1-494e-a6f0-19e233faa92c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:23.144692 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d09ac0-aaa1-494e-a6f0-19e233faa92c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:23.146211 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d09ac0-aaa1-494e-a6f0-19e233faa92c None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:23.147799 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 401/797] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:44:22 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:23.148738 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb8213f-7625-4edc-b245-c77c7d65fd90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:23.149359 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb8213f-7625-4edc-b245-c77c7d65fd90 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:23.149631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb8213f-7625-4edc-b245-c77c7d65fd90 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:23.149897 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb8213f-7625-4edc-b245-c77c7d65fd90 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:23.153793 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d09ac0-aaa1-494e-a6f0-19e233faa92c None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:23.157598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb8213f-7625-4edc-b245-c77c7d65fd90 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:23.159079 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb8213f-7625-4edc-b245-c77c7d65fd90 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:23.170234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-7bb8213f-7625-4edc-b245-c77c7d65fd90 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 30 00:44:23.170234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:23.170234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:23.170234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:23.170234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:23.170234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:23.170234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:23.170234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:23.170234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:23.170234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:23.170234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:23.170234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:23.170234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:44:23.170234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:44:23.170234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:44:23.170234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:44:23.171231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:23.171231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:23.171231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:44:23.171231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:44:23.171231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:44:23.171231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:44:23.171231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:23.171231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:23.171231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:23.171231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:23.171231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:23.171231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:23.171231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:23.171231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:23.171231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:23.171231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:23.171231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:23.171231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:23.172266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:23.172266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:23.172266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:44:23.172266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:44:23.172266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:44:23.172266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:44:23.172266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:44:23.172266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:44:23.172266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 30 00:44:23.172266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:23.172266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 397/798] 10.222.0.16 () {56 vars in 1093 bytes} [Sun Oct 30 00:44:23 2022] GET /identity/v3/users/nova => generated 82 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:44:23.177063 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb8213f-7625-4edc-b245-c77c7d65fd90 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:23.185220 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:23.261884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d09ac0-aaa1-494e-a6f0-19e233faa92c None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:23.278085 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb8213f-7625-4edc-b245-c77c7d65fd90 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:23.295642 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618cb942-c796-41af-829d-73d35e82bec6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:23.296271 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618cb942-c796-41af-829d-73d35e82bec6 None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:23.296545 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618cb942-c796-41af-829d-73d35e82bec6 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:23.296813 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618cb942-c796-41af-829d-73d35e82bec6 None admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:23.297460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618cb942-c796-41af-829d-73d35e82bec6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:23.298970 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618cb942-c796-41af-829d-73d35e82bec6 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:23.311781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c853ef9-6704-4e8d-85e5-2ffdca4f17d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:23.312501 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c853ef9-6704-4e8d-85e5-2ffdca4f17d8 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:23.312906 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c853ef9-6704-4e8d-85e5-2ffdca4f17d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:23.313266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c853ef9-6704-4e8d-85e5-2ffdca4f17d8 None admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:23.314028 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c853ef9-6704-4e8d-85e5-2ffdca4f17d8 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:23.314400 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c853ef9-6704-4e8d-85e5-2ffdca4f17d8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:23.315808 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c853ef9-6704-4e8d-85e5-2ffdca4f17d8 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:23.332011 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 402/799] 10.222.0.16 () {60 vars in 1132 bytes} [Sun Oct 30 00:44:23 2022] POST /identity/v3/users => generated 285 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:44:23.343344 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 398/800] 10.222.0.16 () {56 vars in 1133 bytes} [Sun Oct 30 00:44:23 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:44:23.349952 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618cb942-c796-41af-829d-73d35e82bec6 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:23.443920 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:23.466905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618cb942-c796-41af-829d-73d35e82bec6 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:23.504406 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1824c3-aea6-4404-ac32-9303f2ea13fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:23.505046 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1824c3-aea6-4404-ac32-9303f2ea13fa None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:23.505284 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1824c3-aea6-4404-ac32-9303f2ea13fa None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:23.505551 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1824c3-aea6-4404-ac32-9303f2ea13fa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:23.516350 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1824c3-aea6-4404-ac32-9303f2ea13fa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:23.517937 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1824c3-aea6-4404-ac32-9303f2ea13fa None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:23.532111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-1e1824c3-aea6-4404-ac32-9303f2ea13fa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:44:23.532111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:23.532111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:23.532111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:23.532111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:23.532111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:23.532111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:23.532111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:23.532111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:23.532111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:23.532111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:23.532111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:23.532111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:44:23.532111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:44:23.532111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:44:23.532111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:44:23.533174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:23.533174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:23.533174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:23.533174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:23.533174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:23.533174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:23.533174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:23.533174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:23.533174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:23.533174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:23.533174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:23.533174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:23.533174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:23.533174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:23.533174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:23.533174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:23.533174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:44:23.534086 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:44:23.534086 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:44:23.534086 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:23.534086 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 403/801] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:44:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:23.539269 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c853ef9-6704-4e8d-85e5-2ffdca4f17d8 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:23.548589 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:23.648017 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c853ef9-6704-4e8d-85e5-2ffdca4f17d8 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:23.683097 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eb6494-8e9b-45bd-a72a-0fe6000154c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:23.683699 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eb6494-8e9b-45bd-a72a-0fe6000154c7 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:23.684000 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eb6494-8e9b-45bd-a72a-0fe6000154c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:23.684265 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eb6494-8e9b-45bd-a72a-0fe6000154c7 None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:23.684902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61eb6494-8e9b-45bd-a72a-0fe6000154c7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:23.685198 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61eb6494-8e9b-45bd-a72a-0fe6000154c7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:23.686880 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61eb6494-8e9b-45bd-a72a-0fe6000154c7 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:23.698369 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 399/802] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:44:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:23.704179 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1824c3-aea6-4404-ac32-9303f2ea13fa None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:23.808707 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1824c3-aea6-4404-ac32-9303f2ea13fa None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:23.849750 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea0531a-b3c4-467b-9b4e-764567edc1ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:23.850494 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea0531a-b3c4-467b-9b4e-764567edc1ac None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:23.850743 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea0531a-b3c4-467b-9b4e-764567edc1ac None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:23.851028 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea0531a-b3c4-467b-9b4e-764567edc1ac None admin] PATH_INFO: `/v3/projects/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:23.852954 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:23.859496 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea0531a-b3c4-467b-9b4e-764567edc1ac None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:23.861007 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea0531a-b3c4-467b-9b4e-764567edc1ac None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:23.872353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-dea0531a-b3c4-467b-9b4e-764567edc1ac None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:44:23.872353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:23.872353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:23.872353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:23.872353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:23.872353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:23.872353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:23.872353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:23.872353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:23.872353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:23.872353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:23.872353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:23.872353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:44:23.872353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:44:23.872353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:44:23.873269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:44:23.873269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:23.873269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:23.873269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:23.873269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:23.873269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:23.873269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:23.873269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:23.873269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:23.873269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:23.873269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:23.873269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:23.873269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:23.873269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:23.873269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:23.873269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:23.873269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:23.879598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:44:23.879598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:44:23.879598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:44:23.879598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:44:23.879598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:44:23.879598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:44:23.879598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:44:23.879598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:23.879598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 404/803] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:44:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:23.881477 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eb6494-8e9b-45bd-a72a-0fe6000154c7 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:23.932917 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15765cd-5739-410a-b2ad-514486f94757 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:23.933270 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15765cd-5739-410a-b2ad-514486f94757 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:23.933455 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15765cd-5739-410a-b2ad-514486f94757 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:23.934833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 405/804] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:23.940490 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07197d8e-9e65-44ae-9541-3c8bddfe6d9b None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:23.940765 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07197d8e-9e65-44ae-9541-3c8bddfe6d9b None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:23.941029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07197d8e-9e65-44ae-9541-3c8bddfe6d9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:23.955055 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:23.960021 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-07197d8e-9e65-44ae-9541-3c8bddfe6d9b None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:23.961209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-07197d8e-9e65-44ae-9541-3c8bddfe6d9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:23.999179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eb6494-8e9b-45bd-a72a-0fe6000154c7 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:24.029999 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 406/805] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:24.035962 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fee56a-285d-47e0-9d1f-f45196e4c2fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:24.036802 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fee56a-285d-47e0-9d1f-f45196e4c2fe None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:24.037086 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fee56a-285d-47e0-9d1f-f45196e4c2fe None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:24.037385 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fee56a-285d-47e0-9d1f-f45196e4c2fe None admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:24.038372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fee56a-285d-47e0-9d1f-f45196e4c2fe None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:24.038854 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fee56a-285d-47e0-9d1f-f45196e4c2fe None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:24.040360 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fee56a-285d-47e0-9d1f-f45196e4c2fe None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:24.057748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 400/806] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:44:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:24.058894 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07197d8e-9e65-44ae-9541-3c8bddfe6d9b None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:24.159864 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07197d8e-9e65-44ae-9541-3c8bddfe6d9b None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:24.182795 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:24.195279 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dfef4a-e200-4302-b713-70283bfc3372 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:24.195969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dfef4a-e200-4302-b713-70283bfc3372 None admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:24.196297 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dfef4a-e200-4302-b713-70283bfc3372 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:24.196572 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dfef4a-e200-4302-b713-70283bfc3372 None admin] PATH_INFO: `/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/e7a886715b5b4c1d99f0b37c58cb5417/roles/2aa395773c0a4706897ec0992f3a769a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:24.198844 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26dfef4a-e200-4302-b713-70283bfc3372 None admin] RBAC: Authorizing `identity:create_grant(project_id=658799ed31d2472fb36af4aff0f03aab, user_id=e7a886715b5b4c1d99f0b37c58cb5417, role_id=2aa395773c0a4706897ec0992f3a769a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:24.200304 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26dfef4a-e200-4302-b713-70283bfc3372 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:24.218272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 407/807] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:44:24 2022] PUT /identity/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/e7a886715b5b4c1d99f0b37c58cb5417/roles/2aa395773c0a4706897ec0992f3a769a => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:44:24.302820 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fee56a-285d-47e0-9d1f-f45196e4c2fe None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:24.427151 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fee56a-285d-47e0-9d1f-f45196e4c2fe None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:24.454101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:24.464063 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee1a9cd-e9b3-4556-beed-aaf9a1a24e00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:24.464667 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee1a9cd-e9b3-4556-beed-aaf9a1a24e00 None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:24.464905 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee1a9cd-e9b3-4556-beed-aaf9a1a24e00 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:24.465206 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee1a9cd-e9b3-4556-beed-aaf9a1a24e00 None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:24.465866 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee1a9cd-e9b3-4556-beed-aaf9a1a24e00 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:24.467226 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee1a9cd-e9b3-4556-beed-aaf9a1a24e00 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:24.476727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.sql.core [None req-fee1a9cd-e9b3-4556-beed-aaf9a1a24e00 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 30 00:44:24.476727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 30 00:44:24.476727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [parameters: {'id': 'cb78dad5721d49ff9e839fd2584b34f7', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 30 00:44:24.476727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81494) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 30 00:44:24.492781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-fee1a9cd-e9b3-4556-beed-aaf9a1a24e00 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 30 00:44:24.492781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:24.492781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 30 00:44:24.492781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 30 00:44:24.492781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 30 00:44:24.492781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 30 00:44:24.492781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 30 00:44:24.492781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application result = self._query(query) Oct 30 00:44:24.492781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 30 00:44:24.492781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application conn.query(q) Oct 30 00:44:24.492781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 30 00:44:24.492781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 30 00:44:24.492781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 30 00:44:24.492781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application result.read() Oct 30 00:44:24.493660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 30 00:44:24.493660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 30 00:44:24.493660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 30 00:44:24.493660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 30 00:44:24.493660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 30 00:44:24.493660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 30 00:44:24.493660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 30 00:44:24.493660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 30 00:44:24.493660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 30 00:44:24.493660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:24.493660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 30 00:44:24.493660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:24.493660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:24.493660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 30 00:44:24.493660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 30 00:44:24.493660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 30 00:44:24.493660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 30 00:44:24.493660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 30 00:44:24.494654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application next(self.gen) Oct 30 00:44:24.494654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1068, in _transaction_scope Oct 30 00:44:24.494654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application yield resource Oct 30 00:44:24.494654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 30 00:44:24.494654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application next(self.gen) Oct 30 00:44:24.494654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 670, in _session Oct 30 00:44:24.494654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self.session.rollback() Oct 30 00:44:24.494654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 30 00:44:24.494654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self.force_reraise() Oct 30 00:44:24.494654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 30 00:44:24.494654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise self.value Oct 30 00:44:24.494654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 667, in _session Oct 30 00:44:24.494654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 30 00:44:24.494654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 695, in _end_session_transaction Oct 30 00:44:24.494654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application session.commit() Oct 30 00:44:24.494654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 30 00:44:24.494654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 30 00:44:24.495563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 30 00:44:24.495563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self._prepare_impl() Oct 30 00:44:24.495563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 30 00:44:24.495563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self.session.flush() Oct 30 00:44:24.495563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 30 00:44:24.495563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self._flush(objects) Oct 30 00:44:24.495563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 30 00:44:24.495563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 30 00:44:24.495563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 30 00:44:24.495563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application compat.raise_( Oct 30 00:44:24.495563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 30 00:44:24.495563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception Oct 30 00:44:24.495563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 30 00:44:24.495563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application flush_context.execute() Oct 30 00:44:24.495563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 30 00:44:24.495563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rec.execute(self) Oct 30 00:44:24.495563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 30 00:44:24.498242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 30 00:44:24.498242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 30 00:44:24.498242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 30 00:44:24.498242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 30 00:44:24.498242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 30 00:44:24.498242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 30 00:44:24.498242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 30 00:44:24.498242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 30 00:44:24.498242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 30 00:44:24.498242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 30 00:44:24.498242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 30 00:44:24.498242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 30 00:44:24.498242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 30 00:44:24.498242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 30 00:44:24.498242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 30 00:44:24.500403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 30 00:44:24.500403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception Oct 30 00:44:24.500403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 30 00:44:24.500403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 30 00:44:24.500403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 30 00:44:24.500403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 30 00:44:24.500403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 30 00:44:24.500403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application result = self._query(query) Oct 30 00:44:24.500403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 30 00:44:24.500403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application conn.query(q) Oct 30 00:44:24.500403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 30 00:44:24.500403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 30 00:44:24.500403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 30 00:44:24.500403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application result.read() Oct 30 00:44:24.500403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 30 00:44:24.500403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 30 00:44:24.500403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 30 00:44:24.503215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 30 00:44:24.503215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 30 00:44:24.503215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 30 00:44:24.503215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 30 00:44:24.503215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 30 00:44:24.503215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 30 00:44:24.503215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 30 00:44:24.503215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [parameters: {'id': 'cb78dad5721d49ff9e839fd2584b34f7', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 30 00:44:24.503215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 30 00:44:24.503215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:24.503215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:44:24.503215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:24.503215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:24.503215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 30 00:44:24.503215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 30 00:44:24.505220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 30 00:44:24.505220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 30 00:44:24.505220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 30 00:44:24.505220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:24.505220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:44:24.505220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:24.505220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:24.505220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:24.505220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:24.505220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:24.505220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:24.505220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:24.505220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:24.505220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:24.505220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:24.505220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:24.505220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:24.507252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 30 00:44:24.507252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 30 00:44:24.507252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:24.507252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:24.507252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 30 00:44:24.507252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 30 00:44:24.507252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:24.507252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:24.507252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 30 00:44:24.507252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 30 00:44:24.507252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 30 00:44:24.507252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:24.507252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 401/808] 10.222.0.16 () {60 vars in 1137 bytes} [Sun Oct 30 00:44:24 2022] POST /identity/v3/domains => generated 190 bytes in 192 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:24.508974 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dfef4a-e200-4302-b713-70283bfc3372 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:24.608783 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dfef4a-e200-4302-b713-70283bfc3372 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:24.643898 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6f503c-cfe5-42c2-9d19-4f9f199cbb2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:24.644521 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6f503c-cfe5-42c2-9d19-4f9f199cbb2a None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:24.644794 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6f503c-cfe5-42c2-9d19-4f9f199cbb2a None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:24.645041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6f503c-cfe5-42c2-9d19-4f9f199cbb2a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:24.655676 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6f503c-cfe5-42c2-9d19-4f9f199cbb2a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:24.657127 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6f503c-cfe5-42c2-9d19-4f9f199cbb2a None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:24.672362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-0d6f503c-cfe5-42c2-9d19-4f9f199cbb2a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:44:24.672362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:24.672362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:24.672362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:24.672362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:24.672362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:24.672362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:24.672362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:24.672362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:24.672362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:24.672362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:24.672362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:24.672362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:44:24.672362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:44:24.672362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:44:24.672362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:44:24.673495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:24.673495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:24.673495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:24.673495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:24.673495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:24.673495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:24.673495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:24.673495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:24.673495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:24.673495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:24.673495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:24.673495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:24.673495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:24.673495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:24.673495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:24.673495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:24.673495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:44:24.674569 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:44:24.674569 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:44:24.674569 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:24.674569 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 408/809] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:44:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:24.679778 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee1a9cd-e9b3-4556-beed-aaf9a1a24e00 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:24.784164 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee1a9cd-e9b3-4556-beed-aaf9a1a24e00 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:24.818185 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27dfbb86-5a09-4c8d-b109-96399eac7735 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:24.818903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dfbb86-5a09-4c8d-b109-96399eac7735 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:24.819272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dfbb86-5a09-4c8d-b109-96399eac7735 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:24.819633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dfbb86-5a09-4c8d-b109-96399eac7735 None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:24.820390 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27dfbb86-5a09-4c8d-b109-96399eac7735 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:24.820779 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27dfbb86-5a09-4c8d-b109-96399eac7735 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:24.822226 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27dfbb86-5a09-4c8d-b109-96399eac7735 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:24.834588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 402/810] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:44:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:25.509963 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8e00d5-ed91-4029-bb86-f4555c6254a9 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:25.510546 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8e00d5-ed91-4029-bb86-f4555c6254a9 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:25.510546 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8e00d5-ed91-4029-bb86-f4555c6254a9 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:25.511890 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 409/811] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:25.517584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5be14b-4e47-495a-a56f-5a7967826901 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:25.518055 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5be14b-4e47-495a-a56f-5a7967826901 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:25.518055 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5be14b-4e47-495a-a56f-5a7967826901 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:25.537231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-3a5be14b-4e47-495a-a56f-5a7967826901 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:25.538385 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-3a5be14b-4e47-495a-a56f-5a7967826901 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:25.585367 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:25.608652 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 403/812] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:25 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:25.869951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8e00d5-ed91-4029-bb86-f4555c6254a9 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:25.906934 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:25.973661 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8e00d5-ed91-4029-bb86-f4555c6254a9 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:26.009492 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2af0390-5a3c-4da2-83c4-3056fc284699 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:26.010119 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2af0390-5a3c-4da2-83c4-3056fc284699 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:26.010387 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2af0390-5a3c-4da2-83c4-3056fc284699 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:26.010648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2af0390-5a3c-4da2-83c4-3056fc284699 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:26.022603 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2af0390-5a3c-4da2-83c4-3056fc284699 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:44:26.027415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-c2af0390-5a3c-4da2-83c4-3056fc284699 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:44:26.027415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:26.027415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:26.027415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:26.027415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:26.027415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:26.027415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:26.027415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:26.027415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:26.027415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:26.027415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:26.027415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:26.027415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:44:26.027415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:44:26.027415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:44:26.027415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:44:26.028570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:44:26.028570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:44:26.028570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:44:26.028570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:44:26.028570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:26.028570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:26.028570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:26.028570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:26.028570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:26.028570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:26.028570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:26.028570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:26.028570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:26.028570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:26.028570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:26.028570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:26.029862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:26.029862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:26.029862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:26.029862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:26.029862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:44:26.029862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:44:26.029862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:44:26.029862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:44:26.029862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:44:26.029862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:44:26.029862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:44:26.029862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:26.029862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 410/813] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:44:25 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:26.034761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5be14b-4e47-495a-a56f-5a7967826901 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:26.139556 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5be14b-4e47-495a-a56f-5a7967826901 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:26.173829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5372e0f1-4e8d-4c2d-b0ce-99b5be2f3102 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:26.174427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5372e0f1-4e8d-4c2d-b0ce-99b5be2f3102 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:26.174698 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5372e0f1-4e8d-4c2d-b0ce-99b5be2f3102 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:26.174962 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5372e0f1-4e8d-4c2d-b0ce-99b5be2f3102 None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:26.175616 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5372e0f1-4e8d-4c2d-b0ce-99b5be2f3102 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:26.175953 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5372e0f1-4e8d-4c2d-b0ce-99b5be2f3102 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:26.177304 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5372e0f1-4e8d-4c2d-b0ce-99b5be2f3102 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:26.185568 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 404/814] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:44:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:26.191892 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2af0390-5a3c-4da2-83c4-3056fc284699 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:26.360663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d96299-5bea-4daf-a0c6-64efdaf0bdfd None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:26.360947 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d96299-5bea-4daf-a0c6-64efdaf0bdfd None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:26.361242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d96299-5bea-4daf-a0c6-64efdaf0bdfd None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:26.362979 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 405/815] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:44:26.368095 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be66ab3-47b4-431a-83b7-252f74aec004 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:26.368342 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be66ab3-47b4-431a-83b7-252f74aec004 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:26.368604 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be66ab3-47b4-431a-83b7-252f74aec004 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:26.387004 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-5be66ab3-47b4-431a-83b7-252f74aec004 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:26.388213 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-5be66ab3-47b4-431a-83b7-252f74aec004 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:26.407206 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:26.413939 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2af0390-5a3c-4da2-83c4-3056fc284699 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:26.423109 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:26.448448 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311eed52-c626-47aa-8d03-437be1e32c45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:26.449172 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311eed52-c626-47aa-8d03-437be1e32c45 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:26.449442 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311eed52-c626-47aa-8d03-437be1e32c45 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:26.449704 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311eed52-c626-47aa-8d03-437be1e32c45 None admin] PATH_INFO: `/v3/users/efc2800333154531b1d2f5cc25cc5c85` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:26.456981 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 406/816] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:26.463553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311eed52-c626-47aa-8d03-437be1e32c45 None admin] RBAC: Authorizing `identity:get_user(user_id=efc2800333154531b1d2f5cc25cc5c85)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:26.465033 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311eed52-c626-47aa-8d03-437be1e32c45 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:26.471351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 411/817] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:44:26 2022] GET /identity/v3/users/efc2800333154531b1d2f5cc25cc5c85 => generated 285 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:26.472576 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be66ab3-47b4-431a-83b7-252f74aec004 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:26.568313 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be66ab3-47b4-431a-83b7-252f74aec004 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:26.602860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075a1837-7369-4739-a14c-49568c62e25c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:26.603474 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075a1837-7369-4739-a14c-49568c62e25c None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:26.603734 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075a1837-7369-4739-a14c-49568c62e25c None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:26.604035 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075a1837-7369-4739-a14c-49568c62e25c None admin] PATH_INFO: `/v3/projects/65e5b13c734d464c9f36ce3522fd468e` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:26.605158 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075a1837-7369-4739-a14c-49568c62e25c None admin] RBAC: Authorizing `identity:get_project(project_id=65e5b13c734d464c9f36ce3522fd468e)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:26.606575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075a1837-7369-4739-a14c-49568c62e25c None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:26.608232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 407/818] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:44:26 2022] GET /identity/v3/projects/65e5b13c734d464c9f36ce3522fd468e => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:26.614339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311eed52-c626-47aa-8d03-437be1e32c45 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:26.713277 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311eed52-c626-47aa-8d03-437be1e32c45 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:26.723368 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075a1837-7369-4739-a14c-49568c62e25c None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:26.748451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd95b1b6-a80d-41cf-a97c-1bb897f688de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:26.749105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd95b1b6-a80d-41cf-a97c-1bb897f688de None admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:26.749370 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd95b1b6-a80d-41cf-a97c-1bb897f688de None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:26.749639 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd95b1b6-a80d-41cf-a97c-1bb897f688de None admin] PATH_INFO: `/v3/projects/65e5b13c734d464c9f36ce3522fd468e/users/efc2800333154531b1d2f5cc25cc5c85/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:26.751679 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd95b1b6-a80d-41cf-a97c-1bb897f688de None admin] RBAC: Authorizing `identity:create_grant(project_id=65e5b13c734d464c9f36ce3522fd468e, user_id=efc2800333154531b1d2f5cc25cc5c85, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:26.753115 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd95b1b6-a80d-41cf-a97c-1bb897f688de None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:26.765827 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:26.766880 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 412/819] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:44:26 2022] PUT /identity/v3/projects/65e5b13c734d464c9f36ce3522fd468e/users/efc2800333154531b1d2f5cc25cc5c85/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:44:26.828629 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075a1837-7369-4739-a14c-49568c62e25c None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:26.867313 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eadef2d-0247-4b38-99e0-227e6daf6eeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:26.867974 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eadef2d-0247-4b38-99e0-227e6daf6eeb None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:26.868260 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eadef2d-0247-4b38-99e0-227e6daf6eeb None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:26.868513 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eadef2d-0247-4b38-99e0-227e6daf6eeb None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:26.879886 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eadef2d-0247-4b38-99e0-227e6daf6eeb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:44:26.884701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-8eadef2d-0247-4b38-99e0-227e6daf6eeb None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 30 00:44:26.884701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:26.884701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:26.884701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:26.884701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:26.884701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:26.884701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:26.884701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:26.884701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:26.884701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:26.884701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:26.884701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:26.884701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:44:26.884701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:44:26.884701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:44:26.884701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:44:26.885812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:44:26.885812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:44:26.885812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:44:26.885812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:44:26.885812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:26.885812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:26.885812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:26.885812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:26.885812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:26.885812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:26.885812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:26.885812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:26.885812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:26.885812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:26.885812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:26.885812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:26.887135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:26.887135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:26.887135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:26.887135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:26.887135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:44:26.887135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:44:26.887135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:44:26.887135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:44:26.887135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:44:26.887135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:44:26.887135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 30 00:44:26.887135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:26.887135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 408/820] 10.222.0.16 () {56 vars in 1120 bytes} [Sun Oct 30 00:44:26 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:26.892545 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd95b1b6-a80d-41cf-a97c-1bb897f688de None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:26.910128 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:26.996650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd95b1b6-a80d-41cf-a97c-1bb897f688de None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:27.030371 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c469c220-18ee-4b17-aca0-e1ee25b65a3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:27.030961 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c469c220-18ee-4b17-aca0-e1ee25b65a3c None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:27.031231 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c469c220-18ee-4b17-aca0-e1ee25b65a3c None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:27.031455 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c469c220-18ee-4b17-aca0-e1ee25b65a3c None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:27.032171 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c469c220-18ee-4b17-aca0-e1ee25b65a3c None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:27.032455 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c469c220-18ee-4b17-aca0-e1ee25b65a3c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:27.033945 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c469c220-18ee-4b17-aca0-e1ee25b65a3c None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:27.042404 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 413/821] 10.222.0.16 () {56 vars in 1115 bytes} [Sun Oct 30 00:44:26 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:27.048319 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eadef2d-0247-4b38-99e0-227e6daf6eeb None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:27.147549 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:27.151042 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eadef2d-0247-4b38-99e0-227e6daf6eeb None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:27.185381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6788d348-7987-4306-a1b5-91487bffda32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:27.185991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6788d348-7987-4306-a1b5-91487bffda32 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:27.186591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6788d348-7987-4306-a1b5-91487bffda32 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:27.186856 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6788d348-7987-4306-a1b5-91487bffda32 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:27.197307 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6788d348-7987-4306-a1b5-91487bffda32 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:27.198742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6788d348-7987-4306-a1b5-91487bffda32 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:27.213432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-6788d348-7987-4306-a1b5-91487bffda32 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:44:27.213432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:27.213432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:27.213432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:27.213432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:27.213432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:27.213432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:27.213432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:27.213432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:27.213432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:27.213432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:27.213432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:27.213432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:44:27.213432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:44:27.213432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:44:27.213432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:44:27.214516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:27.214516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:27.214516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:27.214516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:27.214516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:27.214516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:27.214516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:27.214516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:27.214516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:27.214516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:27.214516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:27.214516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:27.214516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:27.214516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:27.214516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:27.214516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:27.214516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:44:27.215517 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:44:27.215517 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:44:27.215517 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:27.219303 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 409/822] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:44:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:27.220381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c469c220-18ee-4b17-aca0-e1ee25b65a3c None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:27.300329 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:27.324376 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c469c220-18ee-4b17-aca0-e1ee25b65a3c None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:27.359054 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f010e6-c603-4930-8ad6-feff664741b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:27.359654 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f010e6-c603-4930-8ad6-feff664741b0 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:27.359960 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f010e6-c603-4930-8ad6-feff664741b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:27.360224 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f010e6-c603-4930-8ad6-feff664741b0 None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:27.360850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f010e6-c603-4930-8ad6-feff664741b0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:27.361332 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f010e6-c603-4930-8ad6-feff664741b0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:27.363785 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f010e6-c603-4930-8ad6-feff664741b0 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:27.375736 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 414/823] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:44:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:27.381593 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6788d348-7987-4306-a1b5-91487bffda32 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:27.486225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6788d348-7987-4306-a1b5-91487bffda32 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:27.522524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f85b6e-daba-4878-b9a6-b1249edd1e76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:27.523143 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f85b6e-daba-4878-b9a6-b1249edd1e76 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:27.523417 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f85b6e-daba-4878-b9a6-b1249edd1e76 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:27.523684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f85b6e-daba-4878-b9a6-b1249edd1e76 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:27.531251 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f85b6e-daba-4878-b9a6-b1249edd1e76 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:27.532707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f85b6e-daba-4878-b9a6-b1249edd1e76 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:27.544050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-76f85b6e-daba-4878-b9a6-b1249edd1e76 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 30 00:44:27.544050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:27.544050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:27.544050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:27.544050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:27.544050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:27.544050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:27.544050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:27.544050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:27.544050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:27.544050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:27.544050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:27.544050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:44:27.544050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:44:27.544050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:44:27.544050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:44:27.549955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:27.549955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:27.549955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:44:27.549955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:44:27.549955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:44:27.549955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:44:27.549955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:27.549955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:27.549955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:27.549955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:27.549955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:27.549955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:27.549955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:27.549955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:27.549955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:27.549955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:27.549955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:27.549955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:27.551290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:27.551290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:27.551290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:44:27.551290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:44:27.551290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:44:27.551290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:44:27.551290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:44:27.551290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:44:27.551290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 30 00:44:27.551290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:27.551290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 410/824] 10.222.0.16 () {56 vars in 1093 bytes} [Sun Oct 30 00:44:27 2022] GET /identity/v3/users/nova => generated 82 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:27.552288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f010e6-c603-4930-8ad6-feff664741b0 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:27.653753 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f010e6-c603-4930-8ad6-feff664741b0 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:27.687198 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907e048b-a2f0-4148-8716-7596df8e5cda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:27.687817 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907e048b-a2f0-4148-8716-7596df8e5cda None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:27.688085 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907e048b-a2f0-4148-8716-7596df8e5cda None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:27.688349 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907e048b-a2f0-4148-8716-7596df8e5cda None admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:27.689055 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907e048b-a2f0-4148-8716-7596df8e5cda None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:27.689408 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907e048b-a2f0-4148-8716-7596df8e5cda None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:27.690888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907e048b-a2f0-4148-8716-7596df8e5cda None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:27.693346 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:27.708132 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 415/825] 10.222.0.16 () {56 vars in 1133 bytes} [Sun Oct 30 00:44:27 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:27.713994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f85b6e-daba-4878-b9a6-b1249edd1e76 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:27.730223 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:27.816147 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f85b6e-daba-4878-b9a6-b1249edd1e76 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:27.850054 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d329e7cf-42c2-4b39-830e-2288998135ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:27.850652 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d329e7cf-42c2-4b39-830e-2288998135ff None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:27.850920 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d329e7cf-42c2-4b39-830e-2288998135ff None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:27.851180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d329e7cf-42c2-4b39-830e-2288998135ff None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:27.918545 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d329e7cf-42c2-4b39-830e-2288998135ff None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:27.920011 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d329e7cf-42c2-4b39-830e-2288998135ff None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:27.933937 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:28.018075 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 411/826] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:44:27 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 30 00:44:28.024019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907e048b-a2f0-4148-8716-7596df8e5cda None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:28.122414 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907e048b-a2f0-4148-8716-7596df8e5cda None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:28.155825 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d578c1dd-c2ba-487b-8e25-15995f36b136 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:28.156440 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d578c1dd-c2ba-487b-8e25-15995f36b136 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:28.156713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d578c1dd-c2ba-487b-8e25-15995f36b136 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:28.156974 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d578c1dd-c2ba-487b-8e25-15995f36b136 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:28.166894 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d578c1dd-c2ba-487b-8e25-15995f36b136 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:28.168367 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d578c1dd-c2ba-487b-8e25-15995f36b136 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:28.182673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-d578c1dd-c2ba-487b-8e25-15995f36b136 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:44:28.182673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:28.182673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:28.182673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:28.182673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:28.182673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:28.182673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:28.182673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:28.182673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:28.182673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:28.182673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:28.182673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:28.182673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:44:28.182673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:44:28.182673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:44:28.182673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:44:28.183842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:28.183842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:28.183842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:28.183842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:28.183842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:28.183842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:28.183842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:28.183842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:28.183842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:28.183842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:28.183842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:28.183842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:28.183842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:28.183842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:28.183842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:28.183842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:28.183842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:44:28.184940 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:44:28.184940 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:44:28.184940 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:28.184940 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 416/827] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:44:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:28.190082 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d329e7cf-42c2-4b39-830e-2288998135ff None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:28.288235 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d329e7cf-42c2-4b39-830e-2288998135ff None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:28.301490 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:28.323213 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a84921-add4-482e-b139-6b528b83a877 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:28.323836 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a84921-add4-482e-b139-6b528b83a877 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:28.324112 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a84921-add4-482e-b139-6b528b83a877 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:28.324374 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a84921-add4-482e-b139-6b528b83a877 None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:28.325005 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a84921-add4-482e-b139-6b528b83a877 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:28.325291 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a84921-add4-482e-b139-6b528b83a877 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:28.326652 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a84921-add4-482e-b139-6b528b83a877 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:28.337764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 412/828] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:44:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:28.343634 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d578c1dd-c2ba-487b-8e25-15995f36b136 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:28.391273 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:28.445199 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d578c1dd-c2ba-487b-8e25-15995f36b136 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:28.479879 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1aed78-5dde-42a6-ae7c-2b6f2d3d5ff3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:28.480497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1aed78-5dde-42a6-ae7c-2b6f2d3d5ff3 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:28.480772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1aed78-5dde-42a6-ae7c-2b6f2d3d5ff3 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:28.481019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1aed78-5dde-42a6-ae7c-2b6f2d3d5ff3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:28.487882 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1aed78-5dde-42a6-ae7c-2b6f2d3d5ff3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:28.489312 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1aed78-5dde-42a6-ae7c-2b6f2d3d5ff3 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:28.500205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-8b1aed78-5dde-42a6-ae7c-2b6f2d3d5ff3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:44:28.500205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:28.500205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:28.500205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:28.500205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:28.500205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:28.500205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:28.500205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:28.500205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:28.500205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:28.500205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:28.500205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:28.500205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:44:28.500205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:44:28.500205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:44:28.501099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:44:28.501099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:28.501099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:28.501099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:28.501099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:28.501099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:28.501099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:28.501099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:28.501099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:28.501099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:28.501099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:28.501099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:28.501099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:28.501099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:28.501099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:28.501099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:28.501099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:28.502116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:44:28.502116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:44:28.502116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:44:28.502116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:44:28.502116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:44:28.502116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:44:28.502116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:44:28.502116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:28.502116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 417/829] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:44:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:28.507494 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a84921-add4-482e-b139-6b528b83a877 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:28.608843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a84921-add4-482e-b139-6b528b83a877 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:28.640845 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c89e34a-3245-486e-8f85-7a46d55c4c8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:28.641606 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c89e34a-3245-486e-8f85-7a46d55c4c8e None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:28.641817 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c89e34a-3245-486e-8f85-7a46d55c4c8e None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:28.642083 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c89e34a-3245-486e-8f85-7a46d55c4c8e None admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:28.642811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c89e34a-3245-486e-8f85-7a46d55c4c8e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:28.643105 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c89e34a-3245-486e-8f85-7a46d55c4c8e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:28.644556 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c89e34a-3245-486e-8f85-7a46d55c4c8e None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:28.657085 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 413/830] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:44:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:28.663442 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1aed78-5dde-42a6-ae7c-2b6f2d3d5ff3 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:28.765855 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1aed78-5dde-42a6-ae7c-2b6f2d3d5ff3 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:28.774111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:28.803286 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c940b14f-d04c-43a9-8be6-19a4aa26402e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:28.803933 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c940b14f-d04c-43a9-8be6-19a4aa26402e None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:28.804206 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c940b14f-d04c-43a9-8be6-19a4aa26402e None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:28.804472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c940b14f-d04c-43a9-8be6-19a4aa26402e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:28.805184 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c940b14f-d04c-43a9-8be6-19a4aa26402e None admin] RBAC: Adding query filter params (role.id=2aa395773c0a4706897ec0992f3a769a, scope.project.id=658799ed31d2472fb36af4aff0f03aab, user.id=e7a886715b5b4c1d99f0b37c58cb5417) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:28.805485 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c940b14f-d04c-43a9-8be6-19a4aa26402e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:28.806900 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c940b14f-d04c-43a9-8be6-19a4aa26402e None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:28.812879 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 418/831] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:44:28 2022] GET /identity/v3/role_assignments?user.id=e7a886715b5b4c1d99f0b37c58cb5417&scope.project.id=658799ed31d2472fb36af4aff0f03aab&role.id=2aa395773c0a4706897ec0992f3a769a => generated 608 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:28.871951 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d629dc5-6c9d-4647-b22e-485ad1a03338 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:28.872515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d629dc5-6c9d-4647-b22e-485ad1a03338 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:28.872515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d629dc5-6c9d-4647-b22e-485ad1a03338 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:28.874051 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 414/832] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:28.880223 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6561096e-f4ef-421e-8601-0c4f27042b57 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:28.880454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6561096e-f4ef-421e-8601-0c4f27042b57 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:28.880761 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6561096e-f4ef-421e-8601-0c4f27042b57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:28.900284 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-6561096e-f4ef-421e-8601-0c4f27042b57 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:28.901376 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-6561096e-f4ef-421e-8601-0c4f27042b57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:28.974245 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 419/833] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:28 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:29.235499 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d629dc5-6c9d-4647-b22e-485ad1a03338 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:29.271536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:29.344248 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d629dc5-6c9d-4647-b22e-485ad1a03338 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:29.380732 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8949fd-c783-49ff-9da3-e1be75655d12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:29.381358 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8949fd-c783-49ff-9da3-e1be75655d12 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:29.381631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8949fd-c783-49ff-9da3-e1be75655d12 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:29.381894 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8949fd-c783-49ff-9da3-e1be75655d12 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:29.391874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8949fd-c783-49ff-9da3-e1be75655d12 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:44:29.396670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-3f8949fd-c783-49ff-9da3-e1be75655d12 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:44:29.396670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:29.396670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:29.396670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:29.396670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:29.396670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:29.396670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:29.396670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:29.396670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:29.396670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:29.396670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:29.396670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:29.396670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:44:29.396670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:44:29.396670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:44:29.396670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:44:29.397788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:44:29.397788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:44:29.397788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:44:29.397788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:44:29.397788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:29.397788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:29.397788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:29.397788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:29.397788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:29.397788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:29.397788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:29.397788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:29.397788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:29.397788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:29.397788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:29.397788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:29.398633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:29.398633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:29.398633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:29.398633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:29.398633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:44:29.398633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:44:29.398633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:44:29.398633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:44:29.398633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:44:29.398633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:44:29.398633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:44:29.398633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:29.398633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 415/834] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:44:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:29.404136 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6561096e-f4ef-421e-8601-0c4f27042b57 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:29.499301 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:29.502624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6561096e-f4ef-421e-8601-0c4f27042b57 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:29.537946 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0d59e4-ffd6-45d0-a236-e14dabae6e4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:29.538562 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0d59e4-ffd6-45d0-a236-e14dabae6e4f None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:29.538836 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0d59e4-ffd6-45d0-a236-e14dabae6e4f None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:29.539109 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0d59e4-ffd6-45d0-a236-e14dabae6e4f None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:29.539795 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0d59e4-ffd6-45d0-a236-e14dabae6e4f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:29.540096 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0d59e4-ffd6-45d0-a236-e14dabae6e4f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:29.541465 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0d59e4-ffd6-45d0-a236-e14dabae6e4f None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:29.549863 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 420/835] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:44:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:29.555964 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8949fd-c783-49ff-9da3-e1be75655d12 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:29.649747 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:29.659345 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8949fd-c783-49ff-9da3-e1be75655d12 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:29.693764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6a2cc3-caa9-4dc1-a0f7-5da4acf76c5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:29.694384 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6a2cc3-caa9-4dc1-a0f7-5da4acf76c5c None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:29.694655 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6a2cc3-caa9-4dc1-a0f7-5da4acf76c5c None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:29.694922 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6a2cc3-caa9-4dc1-a0f7-5da4acf76c5c None admin] PATH_INFO: `/v3/users/efc2800333154531b1d2f5cc25cc5c85` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:29.696092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6a2cc3-caa9-4dc1-a0f7-5da4acf76c5c None admin] RBAC: Authorizing `identity:get_user(user_id=efc2800333154531b1d2f5cc25cc5c85)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:29.697522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6a2cc3-caa9-4dc1-a0f7-5da4acf76c5c None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:29.699106 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 416/836] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:44:29 2022] GET /identity/v3/users/efc2800333154531b1d2f5cc25cc5c85 => generated 285 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:29.705417 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0d59e4-ffd6-45d0-a236-e14dabae6e4f None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:29.808032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0d59e4-ffd6-45d0-a236-e14dabae6e4f None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:29.842607 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5445f086-85d2-40f7-a568-d0036c180fd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:29.843189 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5445f086-85d2-40f7-a568-d0036c180fd9 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:29.843462 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5445f086-85d2-40f7-a568-d0036c180fd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:29.843759 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5445f086-85d2-40f7-a568-d0036c180fd9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:29.873861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:29.922643 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5445f086-85d2-40f7-a568-d0036c180fd9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:29.924239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5445f086-85d2-40f7-a568-d0036c180fd9 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:30.022801 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 421/837] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:44:29 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 30 00:44:30.029011 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6a2cc3-caa9-4dc1-a0f7-5da4acf76c5c None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:30.133270 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6a2cc3-caa9-4dc1-a0f7-5da4acf76c5c None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:30.168260 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a853130-98dd-40a9-b1cc-7206ee9a3bf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:30.168866 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a853130-98dd-40a9-b1cc-7206ee9a3bf5 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:30.169138 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a853130-98dd-40a9-b1cc-7206ee9a3bf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:30.169401 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a853130-98dd-40a9-b1cc-7206ee9a3bf5 None admin] PATH_INFO: `/v3/projects/65e5b13c734d464c9f36ce3522fd468e` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:30.170584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a853130-98dd-40a9-b1cc-7206ee9a3bf5 None admin] RBAC: Authorizing `identity:get_project(project_id=65e5b13c734d464c9f36ce3522fd468e)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:30.172022 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a853130-98dd-40a9-b1cc-7206ee9a3bf5 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:30.173663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 417/838] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:44:30 2022] GET /identity/v3/projects/65e5b13c734d464c9f36ce3522fd468e => generated 311 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:30.180699 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5445f086-85d2-40f7-a568-d0036c180fd9 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:30.255868 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:30.294048 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5445f086-85d2-40f7-a568-d0036c180fd9 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:30.334355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05532c97-faaa-4f9a-bfb5-9169c3d10622 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:30.334951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05532c97-faaa-4f9a-bfb5-9169c3d10622 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:30.335225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05532c97-faaa-4f9a-bfb5-9169c3d10622 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:30.335489 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05532c97-faaa-4f9a-bfb5-9169c3d10622 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:30.336238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05532c97-faaa-4f9a-bfb5-9169c3d10622 None admin] RBAC: Adding query filter params (role.id=35f59686b5d84291a8d9baf19287eb5a, scope.project.id=65e5b13c734d464c9f36ce3522fd468e, user.id=efc2800333154531b1d2f5cc25cc5c85) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:30.336541 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05532c97-faaa-4f9a-bfb5-9169c3d10622 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:30.337989 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05532c97-faaa-4f9a-bfb5-9169c3d10622 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:30.344254 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 422/839] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:44:30 2022] GET /identity/v3/role_assignments?user.id=efc2800333154531b1d2f5cc25cc5c85&scope.project.id=65e5b13c734d464c9f36ce3522fd468e&role.id=35f59686b5d84291a8d9baf19287eb5a => generated 608 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:32.459173 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac36a8d-a84c-4389-8754-5191c639a2a0 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:32.459783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac36a8d-a84c-4389-8754-5191c639a2a0 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:32.459783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac36a8d-a84c-4389-8754-5191c639a2a0 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:32.461075 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 418/840] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:32.467406 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611026d9-134e-4b2e-95ba-018667ca5b15 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:32.468155 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611026d9-134e-4b2e-95ba-018667ca5b15 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:32.468334 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611026d9-134e-4b2e-95ba-018667ca5b15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:32.493695 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-611026d9-134e-4b2e-95ba-018667ca5b15 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:32.494869 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-611026d9-134e-4b2e-95ba-018667ca5b15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:32.567233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 423/841] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:32.843769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac36a8d-a84c-4389-8754-5191c639a2a0 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:32.887410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:32.967179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac36a8d-a84c-4389-8754-5191c639a2a0 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:33.002400 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a2ca96-2994-45ee-8721-f4419c0f8d10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:33.003015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a2ca96-2994-45ee-8721-f4419c0f8d10 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:33.003292 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a2ca96-2994-45ee-8721-f4419c0f8d10 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:33.003538 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a2ca96-2994-45ee-8721-f4419c0f8d10 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:33.004689 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a2ca96-2994-45ee-8721-f4419c0f8d10 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:33.006113 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a2ca96-2994-45ee-8721-f4419c0f8d10 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:33.007663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 419/842] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:44:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:33.014161 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611026d9-134e-4b2e-95ba-018667ca5b15 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:33.076262 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:33.124062 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611026d9-134e-4b2e-95ba-018667ca5b15 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:33.162394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffec57b8-7cf2-42df-8e8e-d2eb9eb42a50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:33.162977 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffec57b8-7cf2-42df-8e8e-d2eb9eb42a50 None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:33.163240 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffec57b8-7cf2-42df-8e8e-d2eb9eb42a50 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:33.163500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffec57b8-7cf2-42df-8e8e-d2eb9eb42a50 None admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:33.164166 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffec57b8-7cf2-42df-8e8e-d2eb9eb42a50 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:33.165575 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffec57b8-7cf2-42df-8e8e-d2eb9eb42a50 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:33.191769 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 424/843] 10.222.0.16 () {60 vars in 1132 bytes} [Sun Oct 30 00:44:33 2022] POST /identity/v3/users => generated 286 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:44:35.313553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b414c4-2d6a-4999-a7c0-699bdfb60731 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:35.314630 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b414c4-2d6a-4999-a7c0-699bdfb60731 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:35.314630 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b414c4-2d6a-4999-a7c0-699bdfb60731 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:35.315818 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 420/844] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:35 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:35.321293 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1960abe-d4f3-423b-9550-54431b8e6be8 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:35.321675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1960abe-d4f3-423b-9550-54431b8e6be8 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:35.321819 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1960abe-d4f3-423b-9550-54431b8e6be8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:35.342245 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-c1960abe-d4f3-423b-9550-54431b8e6be8 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:35.343401 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c1960abe-d4f3-423b-9550-54431b8e6be8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:35.413744 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 425/845] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:35 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:35.681929 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b414c4-2d6a-4999-a7c0-699bdfb60731 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:35.794595 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:35.802712 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b414c4-2d6a-4999-a7c0-699bdfb60731 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:35.838050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fe50b3-076e-4443-847e-4db344dbf9b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:35.838677 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fe50b3-076e-4443-847e-4db344dbf9b0 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:35.838947 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fe50b3-076e-4443-847e-4db344dbf9b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:35.839211 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fe50b3-076e-4443-847e-4db344dbf9b0 None admin] PATH_INFO: `/v3/roles/22e839a60f6f414abd42c577025184a2` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:35.840646 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fe50b3-076e-4443-847e-4db344dbf9b0 None admin] RBAC: Authorizing `identity:get_role(role_id=22e839a60f6f414abd42c577025184a2)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:35.842026 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fe50b3-076e-4443-847e-4db344dbf9b0 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:35.843353 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 421/846] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:44:35 2022] GET /identity/v3/roles/22e839a60f6f414abd42c577025184a2 => generated 223 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:35.852223 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1960abe-d4f3-423b-9550-54431b8e6be8 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:35.895785 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:35.954090 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1960abe-d4f3-423b-9550-54431b8e6be8 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:35.988026 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2418ac54-9484-4f09-a515-37ea9a5ccb4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:35.988639 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2418ac54-9484-4f09-a515-37ea9a5ccb4c None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:35.988910 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2418ac54-9484-4f09-a515-37ea9a5ccb4c None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:35.989186 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2418ac54-9484-4f09-a515-37ea9a5ccb4c None admin] PATH_INFO: `/v3/users/fb0e52edfe4e4a149707826598012417` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:36.002713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2418ac54-9484-4f09-a515-37ea9a5ccb4c None admin] RBAC: Authorizing `identity:get_user(user_id=fb0e52edfe4e4a149707826598012417)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:36.004172 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2418ac54-9484-4f09-a515-37ea9a5ccb4c None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:36.005725 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 426/847] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:44:35 2022] GET /identity/v3/users/fb0e52edfe4e4a149707826598012417 => generated 286 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:36.012343 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fe50b3-076e-4443-847e-4db344dbf9b0 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:36.124912 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fe50b3-076e-4443-847e-4db344dbf9b0 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:36.160984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45f3c8c-2c23-46d1-8e57-374308dbe6c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:36.161603 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45f3c8c-2c23-46d1-8e57-374308dbe6c5 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:36.161874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45f3c8c-2c23-46d1-8e57-374308dbe6c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:36.162143 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45f3c8c-2c23-46d1-8e57-374308dbe6c5 None admin] PATH_INFO: `/v3/projects/65e5b13c734d464c9f36ce3522fd468e` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:36.163317 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45f3c8c-2c23-46d1-8e57-374308dbe6c5 None admin] RBAC: Authorizing `identity:get_project(project_id=65e5b13c734d464c9f36ce3522fd468e)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:36.164741 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45f3c8c-2c23-46d1-8e57-374308dbe6c5 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:36.166378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 422/848] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:44:36 2022] GET /identity/v3/projects/65e5b13c734d464c9f36ce3522fd468e => generated 311 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:36.172727 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2418ac54-9484-4f09-a515-37ea9a5ccb4c None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:36.270884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:36.275110 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2418ac54-9484-4f09-a515-37ea9a5ccb4c None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:36.313448 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2b0e6f-d7f6-4332-a1ce-e0f893b7705a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:36.314065 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2b0e6f-d7f6-4332-a1ce-e0f893b7705a None admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:36.314460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2b0e6f-d7f6-4332-a1ce-e0f893b7705a None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:36.314714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2b0e6f-d7f6-4332-a1ce-e0f893b7705a None admin] PATH_INFO: `/v3/projects/65e5b13c734d464c9f36ce3522fd468e/users/fb0e52edfe4e4a149707826598012417/roles/22e839a60f6f414abd42c577025184a2` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:36.317024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2b0e6f-d7f6-4332-a1ce-e0f893b7705a None admin] RBAC: Authorizing `identity:create_grant(project_id=65e5b13c734d464c9f36ce3522fd468e, user_id=fb0e52edfe4e4a149707826598012417, role_id=22e839a60f6f414abd42c577025184a2)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:36.318552 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2b0e6f-d7f6-4332-a1ce-e0f893b7705a None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:36.334482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 427/849] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:44:36 2022] PUT /identity/v3/projects/65e5b13c734d464c9f36ce3522fd468e/users/fb0e52edfe4e4a149707826598012417/roles/22e839a60f6f414abd42c577025184a2 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:44:38.443786 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd92a146-e39a-4cfe-8247-847ac014eff9 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:38.444611 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd92a146-e39a-4cfe-8247-847ac014eff9 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:38.444720 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd92a146-e39a-4cfe-8247-847ac014eff9 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:38.446828 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 423/850] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:38 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:38.453213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441341b0-2158-4e67-aa14-4aa770358622 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:38.453964 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441341b0-2158-4e67-aa14-4aa770358622 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:38.454087 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441341b0-2158-4e67-aa14-4aa770358622 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:38.474501 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-441341b0-2158-4e67-aa14-4aa770358622 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:38.475664 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-441341b0-2158-4e67-aa14-4aa770358622 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:38.546140 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 428/851] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:38.822285 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd92a146-e39a-4cfe-8247-847ac014eff9 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:38.864103 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:38.938781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd92a146-e39a-4cfe-8247-847ac014eff9 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:38.975233 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d31d9a4-1b42-49f1-8975-aee8c9dcf642 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:38.975869 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d31d9a4-1b42-49f1-8975-aee8c9dcf642 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:38.976137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d31d9a4-1b42-49f1-8975-aee8c9dcf642 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:38.976400 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d31d9a4-1b42-49f1-8975-aee8c9dcf642 None admin] PATH_INFO: `/v3/roles/22e839a60f6f414abd42c577025184a2` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:38.977811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d31d9a4-1b42-49f1-8975-aee8c9dcf642 None admin] RBAC: Authorizing `identity:get_role(role_id=22e839a60f6f414abd42c577025184a2)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:38.979164 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d31d9a4-1b42-49f1-8975-aee8c9dcf642 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:38.980500 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 424/852] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:44:38 2022] GET /identity/v3/roles/22e839a60f6f414abd42c577025184a2 => generated 223 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:38.989983 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441341b0-2158-4e67-aa14-4aa770358622 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:39.087942 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:39.099906 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441341b0-2158-4e67-aa14-4aa770358622 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:39.136845 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405762f9-599b-4955-8069-a61b298edf83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:39.137455 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405762f9-599b-4955-8069-a61b298edf83 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:39.137729 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405762f9-599b-4955-8069-a61b298edf83 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:39.137992 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405762f9-599b-4955-8069-a61b298edf83 None admin] PATH_INFO: `/v3/users/fb0e52edfe4e4a149707826598012417` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:39.139216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405762f9-599b-4955-8069-a61b298edf83 None admin] RBAC: Authorizing `identity:get_user(user_id=fb0e52edfe4e4a149707826598012417)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:39.140658 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405762f9-599b-4955-8069-a61b298edf83 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:39.142255 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 429/853] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:44:38 2022] GET /identity/v3/users/fb0e52edfe4e4a149707826598012417 => generated 286 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:39.148242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d31d9a4-1b42-49f1-8975-aee8c9dcf642 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:39.260899 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:39.264252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d31d9a4-1b42-49f1-8975-aee8c9dcf642 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:39.302086 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86846b0a-d659-4921-9f6f-6e3ade725289 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:39.302679 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86846b0a-d659-4921-9f6f-6e3ade725289 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:39.302951 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86846b0a-d659-4921-9f6f-6e3ade725289 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:39.303216 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86846b0a-d659-4921-9f6f-6e3ade725289 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:39.373179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86846b0a-d659-4921-9f6f-6e3ade725289 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:39.374621 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86846b0a-d659-4921-9f6f-6e3ade725289 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:39.471505 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:39.475339 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 425/854] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:44:39 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 30 00:44:39.481459 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405762f9-599b-4955-8069-a61b298edf83 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:39.586109 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405762f9-599b-4955-8069-a61b298edf83 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:39.621216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aaa9205-922b-49d8-8f8f-b86834a32f2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:39.621759 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aaa9205-922b-49d8-8f8f-b86834a32f2b None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:39.622025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aaa9205-922b-49d8-8f8f-b86834a32f2b None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:39.622293 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aaa9205-922b-49d8-8f8f-b86834a32f2b None admin] PATH_INFO: `/v3/projects/65e5b13c734d464c9f36ce3522fd468e` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:39.623603 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aaa9205-922b-49d8-8f8f-b86834a32f2b None admin] RBAC: Authorizing `identity:get_project(project_id=65e5b13c734d464c9f36ce3522fd468e)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:39.625045 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aaa9205-922b-49d8-8f8f-b86834a32f2b None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:39.626745 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 430/855] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:44:39 2022] GET /identity/v3/projects/65e5b13c734d464c9f36ce3522fd468e => generated 311 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:39.633277 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86846b0a-d659-4921-9f6f-6e3ade725289 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:39.741578 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86846b0a-d659-4921-9f6f-6e3ade725289 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:39.776694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b9602a-1518-48d9-a660-de1c9db43272 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:39.777282 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b9602a-1518-48d9-a660-de1c9db43272 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:39.777552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b9602a-1518-48d9-a660-de1c9db43272 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:39.777815 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b9602a-1518-48d9-a660-de1c9db43272 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:39.778536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b9602a-1518-48d9-a660-de1c9db43272 None admin] RBAC: Adding query filter params (role.id=22e839a60f6f414abd42c577025184a2, scope.project.id=65e5b13c734d464c9f36ce3522fd468e, user.id=fb0e52edfe4e4a149707826598012417) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:39.778834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b9602a-1518-48d9-a660-de1c9db43272 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:39.780285 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b9602a-1518-48d9-a660-de1c9db43272 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:39.786119 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 426/856] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:44:39 2022] GET /identity/v3/role_assignments?user.id=fb0e52edfe4e4a149707826598012417&scope.project.id=65e5b13c734d464c9f36ce3522fd468e&role.id=22e839a60f6f414abd42c577025184a2 => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:41.894219 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8421a7e-ea03-49ed-97ed-8e2168a8fc21 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:41.894779 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8421a7e-ea03-49ed-97ed-8e2168a8fc21 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:41.894779 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8421a7e-ea03-49ed-97ed-8e2168a8fc21 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:41.896146 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 431/857] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:41.902406 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca462e26-e9fb-481b-ae1e-67c665649d74 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:41.903106 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca462e26-e9fb-481b-ae1e-67c665649d74 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:41.903250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca462e26-e9fb-481b-ae1e-67c665649d74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:41.925758 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-ca462e26-e9fb-481b-ae1e-67c665649d74 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:41.926937 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-ca462e26-e9fb-481b-ae1e-67c665649d74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:41.990407 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:42.000074 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 427/858] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:42.265882 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8421a7e-ea03-49ed-97ed-8e2168a8fc21 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:42.297955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:42.378069 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8421a7e-ea03-49ed-97ed-8e2168a8fc21 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:42.414230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4fbd8ed-03c1-42da-b495-4654a902a94d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:42.414842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fbd8ed-03c1-42da-b495-4654a902a94d None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:42.415169 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fbd8ed-03c1-42da-b495-4654a902a94d None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:42.415434 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fbd8ed-03c1-42da-b495-4654a902a94d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:42.416615 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4fbd8ed-03c1-42da-b495-4654a902a94d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:42.418065 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4fbd8ed-03c1-42da-b495-4654a902a94d None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:42.419608 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 432/859] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:44:42 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:42.426030 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca462e26-e9fb-481b-ae1e-67c665649d74 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:42.531290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca462e26-e9fb-481b-ae1e-67c665649d74 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:42.566875 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993f1ac3-4bb0-456b-87c5-70a4e86d7f0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:42.567452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993f1ac3-4bb0-456b-87c5-70a4e86d7f0a None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:42.567751 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993f1ac3-4bb0-456b-87c5-70a4e86d7f0a None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:42.568011 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993f1ac3-4bb0-456b-87c5-70a4e86d7f0a None admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:42.568641 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993f1ac3-4bb0-456b-87c5-70a4e86d7f0a None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:42.570046 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993f1ac3-4bb0-456b-87c5-70a4e86d7f0a None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:42.602945 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 428/860] 10.222.0.16 () {60 vars in 1141 bytes} [Sun Oct 30 00:44:42 2022] POST /identity/v3/projects => generated 311 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:44:44.741049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92407df-70e2-4fac-b75d-f92c6b7fa924 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:44.741586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92407df-70e2-4fac-b75d-f92c6b7fa924 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:44.741586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92407df-70e2-4fac-b75d-f92c6b7fa924 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:44.742819 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 433/861] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:44.749152 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbab202b-fcd9-445a-8ddc-096ce9fe7739 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:44.749858 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbab202b-fcd9-445a-8ddc-096ce9fe7739 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:44.750007 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbab202b-fcd9-445a-8ddc-096ce9fe7739 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:44.770555 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-dbab202b-fcd9-445a-8ddc-096ce9fe7739 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:44.771754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-dbab202b-fcd9-445a-8ddc-096ce9fe7739 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:44.821091 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:44.847660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 429/862] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:45.115471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92407df-70e2-4fac-b75d-f92c6b7fa924 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:45.214924 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:45.228700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92407df-70e2-4fac-b75d-f92c6b7fa924 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:45.265266 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d2ef39-9dc8-4dc5-ae45-3b91a6237bd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:45.265882 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d2ef39-9dc8-4dc5-ae45-3b91a6237bd6 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:45.266152 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d2ef39-9dc8-4dc5-ae45-3b91a6237bd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:45.266421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d2ef39-9dc8-4dc5-ae45-3b91a6237bd6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:45.267598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d2ef39-9dc8-4dc5-ae45-3b91a6237bd6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:45.269065 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d2ef39-9dc8-4dc5-ae45-3b91a6237bd6 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:45.270605 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 434/863] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:44:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:45.277461 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbab202b-fcd9-445a-8ddc-096ce9fe7739 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:45.391271 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbab202b-fcd9-445a-8ddc-096ce9fe7739 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:45.433549 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37eb3934-ba0d-4b20-b1a4-1adb1c458ece None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:45.434149 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37eb3934-ba0d-4b20-b1a4-1adb1c458ece None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:45.434422 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37eb3934-ba0d-4b20-b1a4-1adb1c458ece None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:45.434686 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37eb3934-ba0d-4b20-b1a4-1adb1c458ece None admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:45.435320 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37eb3934-ba0d-4b20-b1a4-1adb1c458ece None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:45.437094 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37eb3934-ba0d-4b20-b1a4-1adb1c458ece None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:45.459978 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 430/864] 10.222.0.16 () {60 vars in 1132 bytes} [Sun Oct 30 00:44:45 2022] POST /identity/v3/users => generated 286 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:44:47.614674 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bda30b-8c08-4a80-9dc2-18df3761d2c9 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:47.615514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bda30b-8c08-4a80-9dc2-18df3761d2c9 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:47.615514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bda30b-8c08-4a80-9dc2-18df3761d2c9 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:47.617476 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 435/865] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:47 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:47.624055 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be1dd39-3f07-45f0-b985-2d16dae24433 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:47.624721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be1dd39-3f07-45f0-b985-2d16dae24433 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:47.624893 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be1dd39-3f07-45f0-b985-2d16dae24433 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:47.650737 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-0be1dd39-3f07-45f0-b985-2d16dae24433 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:47.651941 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-0be1dd39-3f07-45f0-b985-2d16dae24433 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:47.679234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:47.724674 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 431/866] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:47.993460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bda30b-8c08-4a80-9dc2-18df3761d2c9 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:48.101592 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bda30b-8c08-4a80-9dc2-18df3761d2c9 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:48.139083 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d9329a-99e7-42cd-81f0-8a1447cba323 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:48.139697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d9329a-99e7-42cd-81f0-8a1447cba323 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:48.139997 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d9329a-99e7-42cd-81f0-8a1447cba323 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:48.140266 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d9329a-99e7-42cd-81f0-8a1447cba323 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:48.151026 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d9329a-99e7-42cd-81f0-8a1447cba323 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:44:48.155803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-73d9329a-99e7-42cd-81f0-8a1447cba323 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:44:48.155803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:48.155803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:48.155803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:48.155803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:48.155803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:48.155803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:48.155803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:48.155803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:48.155803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:48.155803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:48.155803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:48.155803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:44:48.155803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:44:48.155803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:44:48.155803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:44:48.156840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:44:48.156840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:44:48.156840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:44:48.156840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:44:48.156840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:48.156840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:48.156840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:48.156840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:48.156840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:48.156840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:48.156840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:48.156840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:48.156840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:48.156840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:48.156840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:48.156840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:48.157691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:48.157691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:48.157691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:48.157691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:48.157691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:44:48.157691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:44:48.157691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:44:48.157691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:44:48.157691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:44:48.157691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:44:48.157691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:44:48.157691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:44:48.157691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 436/867] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:44:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:48.165070 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be1dd39-3f07-45f0-b985-2d16dae24433 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:48.266349 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be1dd39-3f07-45f0-b985-2d16dae24433 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:48.304006 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1aaedb1-64dc-4a05-b0c1-38065461ebdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:48.304599 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1aaedb1-64dc-4a05-b0c1-38065461ebdd None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:48.304870 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1aaedb1-64dc-4a05-b0c1-38065461ebdd None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:48.305119 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1aaedb1-64dc-4a05-b0c1-38065461ebdd None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:48.305776 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1aaedb1-64dc-4a05-b0c1-38065461ebdd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:48.306073 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1aaedb1-64dc-4a05-b0c1-38065461ebdd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:48.307429 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1aaedb1-64dc-4a05-b0c1-38065461ebdd None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:48.315880 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 432/868] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:44:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:48.324970 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d9329a-99e7-42cd-81f0-8a1447cba323 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:48.354131 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:48.432924 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d9329a-99e7-42cd-81f0-8a1447cba323 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:48.467867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f653f1-19a7-4331-bee6-07c73c44ed64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:48.468479 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f653f1-19a7-4331-bee6-07c73c44ed64 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:48.468750 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f653f1-19a7-4331-bee6-07c73c44ed64 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:48.469019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f653f1-19a7-4331-bee6-07c73c44ed64 None admin] PATH_INFO: `/v3/users/e119184f31254cffa880631bc2aa9f07` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:48.481789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f653f1-19a7-4331-bee6-07c73c44ed64 None admin] RBAC: Authorizing `identity:get_user(user_id=e119184f31254cffa880631bc2aa9f07)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:48.483218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f653f1-19a7-4331-bee6-07c73c44ed64 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:48.484828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 437/869] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:44:48 2022] GET /identity/v3/users/e119184f31254cffa880631bc2aa9f07 => generated 286 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:48.491184 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1aaedb1-64dc-4a05-b0c1-38065461ebdd None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:48.519147 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:48.610742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1aaedb1-64dc-4a05-b0c1-38065461ebdd None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:48.645192 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46618b1f-0bdb-4a81-a3ba-ae01a5801653 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:48.645768 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46618b1f-0bdb-4a81-a3ba-ae01a5801653 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:48.646057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46618b1f-0bdb-4a81-a3ba-ae01a5801653 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:48.646322 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46618b1f-0bdb-4a81-a3ba-ae01a5801653 None admin] PATH_INFO: `/v3/projects/1eb5c4f8e0fb494caf7c78e7c80062ce` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:48.647486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46618b1f-0bdb-4a81-a3ba-ae01a5801653 None admin] RBAC: Authorizing `identity:get_project(project_id=1eb5c4f8e0fb494caf7c78e7c80062ce)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:48.649000 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46618b1f-0bdb-4a81-a3ba-ae01a5801653 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:48.650666 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 433/870] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:44:48 2022] GET /identity/v3/projects/1eb5c4f8e0fb494caf7c78e7c80062ce => generated 311 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:48.659946 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f653f1-19a7-4331-bee6-07c73c44ed64 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:48.741158 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:48.760852 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f653f1-19a7-4331-bee6-07c73c44ed64 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:48.795147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127f93c6-9084-41d4-ac59-e638c2f93f8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:48.795749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127f93c6-9084-41d4-ac59-e638c2f93f8f None admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:48.796017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127f93c6-9084-41d4-ac59-e638c2f93f8f None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:48.796281 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127f93c6-9084-41d4-ac59-e638c2f93f8f None admin] PATH_INFO: `/v3/projects/1eb5c4f8e0fb494caf7c78e7c80062ce/users/e119184f31254cffa880631bc2aa9f07/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:48.798242 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127f93c6-9084-41d4-ac59-e638c2f93f8f None admin] RBAC: Authorizing `identity:create_grant(project_id=1eb5c4f8e0fb494caf7c78e7c80062ce, user_id=e119184f31254cffa880631bc2aa9f07, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:48.799702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127f93c6-9084-41d4-ac59-e638c2f93f8f None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:48.816045 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 438/871] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:44:48 2022] PUT /identity/v3/projects/1eb5c4f8e0fb494caf7c78e7c80062ce/users/e119184f31254cffa880631bc2aa9f07/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:44:50.897852 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb31889-f1d0-4b69-ad5f-afd1d2bb9234 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:50.898428 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb31889-f1d0-4b69-ad5f-afd1d2bb9234 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:50.898428 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb31889-f1d0-4b69-ad5f-afd1d2bb9234 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:50.899830 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 434/872] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:50.906397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c847214-881b-4e51-93cb-e5a42c295e68 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:50.907108 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c847214-881b-4e51-93cb-e5a42c295e68 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:50.907275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c847214-881b-4e51-93cb-e5a42c295e68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:50.927598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-5c847214-881b-4e51-93cb-e5a42c295e68 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:50.928847 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-5c847214-881b-4e51-93cb-e5a42c295e68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:51.006420 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 439/873] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:50 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:51.278952 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb31889-f1d0-4b69-ad5f-afd1d2bb9234 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:51.390995 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:51.402102 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb31889-f1d0-4b69-ad5f-afd1d2bb9234 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:51.438341 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021cd1b3-9c2e-4ce8-b834-c14a102c8783 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:51.439023 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021cd1b3-9c2e-4ce8-b834-c14a102c8783 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:51.439292 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021cd1b3-9c2e-4ce8-b834-c14a102c8783 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:51.439557 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021cd1b3-9c2e-4ce8-b834-c14a102c8783 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:51.450162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021cd1b3-9c2e-4ce8-b834-c14a102c8783 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:44:51.454868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-021cd1b3-9c2e-4ce8-b834-c14a102c8783 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:44:51.454868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:44:51.454868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:44:51.454868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:44:51.454868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:44:51.454868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:44:51.454868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:44:51.454868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:44:51.454868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:44:51.454868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:44:51.454868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:44:51.454868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:44:51.454868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:44:51.454868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:44:51.454868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:44:51.454868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:44:51.455988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:44:51.455988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:44:51.455988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:44:51.455988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:44:51.455988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:44:51.455988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:44:51.455988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:44:51.455988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:44:51.455988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:44:51.455988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:44:51.455988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:44:51.455988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:44:51.455988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:44:51.455988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:44:51.455988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:44:51.455988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:44:51.456874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:44:51.456874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:44:51.456874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:44:51.456874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:44:51.456874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:44:51.456874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:44:51.456874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:44:51.456874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:44:51.456874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:44:51.456874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:44:51.456874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:44:51.456874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:44:51.456874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 435/874] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:44:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:44:51.462552 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c847214-881b-4e51-93cb-e5a42c295e68 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:51.535815 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:51.571051 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c847214-881b-4e51-93cb-e5a42c295e68 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:51.606619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8353df-3554-4052-9c08-4ab8be9bf394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:51.607226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8353df-3554-4052-9c08-4ab8be9bf394 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:51.607614 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8353df-3554-4052-9c08-4ab8be9bf394 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:51.607916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8353df-3554-4052-9c08-4ab8be9bf394 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:51.608573 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8353df-3554-4052-9c08-4ab8be9bf394 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:51.608865 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8353df-3554-4052-9c08-4ab8be9bf394 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:51.610209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8353df-3554-4052-9c08-4ab8be9bf394 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:51.618655 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 440/875] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:44:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:51.625631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021cd1b3-9c2e-4ce8-b834-c14a102c8783 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:51.745005 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021cd1b3-9c2e-4ce8-b834-c14a102c8783 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:51.780872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94ac044-ec23-43aa-946a-4e6b181ceb22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:51.781503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94ac044-ec23-43aa-946a-4e6b181ceb22 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:51.781778 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94ac044-ec23-43aa-946a-4e6b181ceb22 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:51.782047 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94ac044-ec23-43aa-946a-4e6b181ceb22 None admin] PATH_INFO: `/v3/users/e119184f31254cffa880631bc2aa9f07` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:51.783262 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94ac044-ec23-43aa-946a-4e6b181ceb22 None admin] RBAC: Authorizing `identity:get_user(user_id=e119184f31254cffa880631bc2aa9f07)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:51.784728 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e94ac044-ec23-43aa-946a-4e6b181ceb22 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:51.786577 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 436/876] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:44:51 2022] GET /identity/v3/users/e119184f31254cffa880631bc2aa9f07 => generated 286 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:51.792308 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8353df-3554-4052-9c08-4ab8be9bf394 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:51.889746 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8353df-3554-4052-9c08-4ab8be9bf394 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:51.918748 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:51.924884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4165239-0a65-4df6-abec-7c61e14822ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:51.925520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4165239-0a65-4df6-abec-7c61e14822ca None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:51.925791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4165239-0a65-4df6-abec-7c61e14822ca None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:51.926055 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4165239-0a65-4df6-abec-7c61e14822ca None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:51.993612 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4165239-0a65-4df6-abec-7c61e14822ca None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:51.995053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4165239-0a65-4df6-abec-7c61e14822ca None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:52.101084 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 441/877] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:44:51 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 30 00:44:52.107588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94ac044-ec23-43aa-946a-4e6b181ceb22 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:52.133739 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:52.224032 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94ac044-ec23-43aa-946a-4e6b181ceb22 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:52.261248 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0c02df-1797-4005-a902-489306088272 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:52.261862 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0c02df-1797-4005-a902-489306088272 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:52.262130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0c02df-1797-4005-a902-489306088272 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:52.262397 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0c02df-1797-4005-a902-489306088272 None admin] PATH_INFO: `/v3/projects/1eb5c4f8e0fb494caf7c78e7c80062ce` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:52.263537 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0c02df-1797-4005-a902-489306088272 None admin] RBAC: Authorizing `identity:get_project(project_id=1eb5c4f8e0fb494caf7c78e7c80062ce)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:52.264996 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0c02df-1797-4005-a902-489306088272 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:52.266601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 437/878] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:44:52 2022] GET /identity/v3/projects/1eb5c4f8e0fb494caf7c78e7c80062ce => generated 311 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:52.273861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4165239-0a65-4df6-abec-7c61e14822ca None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:52.309697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:52.385492 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4165239-0a65-4df6-abec-7c61e14822ca None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:52.422284 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c78b5f7-5c85-4d34-824e-0a151ac69847 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:52.422885 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c78b5f7-5c85-4d34-824e-0a151ac69847 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:52.423155 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c78b5f7-5c85-4d34-824e-0a151ac69847 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:52.423416 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c78b5f7-5c85-4d34-824e-0a151ac69847 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:52.424166 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c78b5f7-5c85-4d34-824e-0a151ac69847 None admin] RBAC: Adding query filter params (role.id=35f59686b5d84291a8d9baf19287eb5a, scope.project.id=1eb5c4f8e0fb494caf7c78e7c80062ce, user.id=e119184f31254cffa880631bc2aa9f07) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:44:52.424464 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c78b5f7-5c85-4d34-824e-0a151ac69847 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:52.425889 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c78b5f7-5c85-4d34-824e-0a151ac69847 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:52.432543 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 442/879] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:44:52 2022] GET /identity/v3/role_assignments?user.id=e119184f31254cffa880631bc2aa9f07&scope.project.id=1eb5c4f8e0fb494caf7c78e7c80062ce&role.id=35f59686b5d84291a8d9baf19287eb5a => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:54.549112 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd03ec2-1622-4a8c-b9ce-e2a1b613f6b2 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:54.549730 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd03ec2-1622-4a8c-b9ce-e2a1b613f6b2 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:54.549824 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd03ec2-1622-4a8c-b9ce-e2a1b613f6b2 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:54.551152 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 438/880] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:54.557550 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1239f3ca-08d0-433c-9817-6383c0142c99 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:54.558306 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1239f3ca-08d0-433c-9817-6383c0142c99 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:54.558451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1239f3ca-08d0-433c-9817-6383c0142c99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:54.579314 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-1239f3ca-08d0-433c-9817-6383c0142c99 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:54.580573 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-1239f3ca-08d0-433c-9817-6383c0142c99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:54.651809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 443/881] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:54.916293 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd03ec2-1622-4a8c-b9ce-e2a1b613f6b2 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:55.011778 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:55.025869 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd03ec2-1622-4a8c-b9ce-e2a1b613f6b2 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:55.060462 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c397099b-ae8a-4180-ac59-21b5dde3102b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:55.061048 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c397099b-ae8a-4180-ac59-21b5dde3102b None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:55.061317 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c397099b-ae8a-4180-ac59-21b5dde3102b None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:55.061565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c397099b-ae8a-4180-ac59-21b5dde3102b None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:55.062592 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c397099b-ae8a-4180-ac59-21b5dde3102b None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:55.064009 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c397099b-ae8a-4180-ac59-21b5dde3102b None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:55.084930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 439/882] 10.222.0.16 () {60 vars in 1138 bytes} [Sun Oct 30 00:44:54 2022] POST /identity/v3/domains => generated 267 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:44:57.213422 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b513afce-6fd7-4c93-b652-05018493746a None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:57.214009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b513afce-6fd7-4c93-b652-05018493746a None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:57.214009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b513afce-6fd7-4c93-b652-05018493746a None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:57.215297 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 444/883] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:44:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:44:57.221868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b881ab0d-4ee7-422d-ac49-6eeda8f9929d None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:57.222611 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b881ab0d-4ee7-422d-ac49-6eeda8f9929d None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:57.222773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b881ab0d-4ee7-422d-ac49-6eeda8f9929d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:57.244321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-b881ab0d-4ee7-422d-ac49-6eeda8f9929d None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:44:57.245480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b881ab0d-4ee7-422d-ac49-6eeda8f9929d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:44:57.327515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 440/884] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:44:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:44:57.601233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b513afce-6fd7-4c93-b652-05018493746a None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:57.631999 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:57.706858 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b513afce-6fd7-4c93-b652-05018493746a None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:57.745774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72cb22c-87f8-475d-bb2c-b6d442c1c7a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:57.746394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72cb22c-87f8-475d-bb2c-b6d442c1c7a8 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:57.746668 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72cb22c-87f8-475d-bb2c-b6d442c1c7a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:57.746934 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72cb22c-87f8-475d-bb2c-b6d442c1c7a8 None admin] PATH_INFO: `/v3/domains/5c02ae63e5b749049d81dff50b1c7898` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:57.757879 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72cb22c-87f8-475d-bb2c-b6d442c1c7a8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=5c02ae63e5b749049d81dff50b1c7898)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:57.759316 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72cb22c-87f8-475d-bb2c-b6d442c1c7a8 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:57.760908 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 445/885] 10.222.0.16 () {56 vars in 1183 bytes} [Sun Oct 30 00:44:57 2022] GET /identity/v3/domains/5c02ae63e5b749049d81dff50b1c7898 => generated 267 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:44:57.767690 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b881ab0d-4ee7-422d-ac49-6eeda8f9929d None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:44:57.856575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:44:57.879310 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b881ab0d-4ee7-422d-ac49-6eeda8f9929d None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:44:57.914086 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83ec414-a7c6-4d71-8b38-50eb32c4942f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:44:57.914671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83ec414-a7c6-4d71-8b38-50eb32c4942f None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:44:57.914940 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83ec414-a7c6-4d71-8b38-50eb32c4942f None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:44:57.915201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83ec414-a7c6-4d71-8b38-50eb32c4942f None admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:44:57.915862 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83ec414-a7c6-4d71-8b38-50eb32c4942f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:44:57.917280 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83ec414-a7c6-4d71-8b38-50eb32c4942f None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:44:57.951017 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 441/886] 10.222.0.16 () {60 vars in 1141 bytes} [Sun Oct 30 00:44:57 2022] POST /identity/v3/projects => generated 361 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:45:00.114399 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3309fbfd-d5a5-4e0c-8007-0e7206be1b60 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:45:00.115289 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3309fbfd-d5a5-4e0c-8007-0e7206be1b60 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:45:00.115289 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3309fbfd-d5a5-4e0c-8007-0e7206be1b60 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:45:00.117508 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 446/887] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:45:00 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:45:00.123995 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1bd10e-0d30-4a4f-86e4-96cd0a5b527e None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:45:00.124439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1bd10e-0d30-4a4f-86e4-96cd0a5b527e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:45:00.124531 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1bd10e-0d30-4a4f-86e4-96cd0a5b527e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:45:00.151089 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-4a1bd10e-0d30-4a4f-86e4-96cd0a5b527e None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:45:00.152330 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-4a1bd10e-0d30-4a4f-86e4-96cd0a5b527e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:45:00.229219 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 442/888] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:45:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:45:00.501050 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3309fbfd-d5a5-4e0c-8007-0e7206be1b60 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:45:00.597881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:45:00.614643 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3309fbfd-d5a5-4e0c-8007-0e7206be1b60 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:45:00.648880 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403184f9-bfa7-49ff-a875-55977058ddef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:45:00.649493 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403184f9-bfa7-49ff-a875-55977058ddef None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:45:00.649767 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403184f9-bfa7-49ff-a875-55977058ddef None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:45:00.650029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403184f9-bfa7-49ff-a875-55977058ddef None admin] PATH_INFO: `/v3/domains/5c02ae63e5b749049d81dff50b1c7898` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:45:00.651231 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403184f9-bfa7-49ff-a875-55977058ddef None admin] RBAC: Authorizing `identity:get_domain(domain_id=5c02ae63e5b749049d81dff50b1c7898)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:45:00.652676 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403184f9-bfa7-49ff-a875-55977058ddef None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:45:00.654217 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 447/889] 10.222.0.16 () {56 vars in 1183 bytes} [Sun Oct 30 00:45:00 2022] GET /identity/v3/domains/5c02ae63e5b749049d81dff50b1c7898 => generated 267 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:45:00.660417 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1bd10e-0d30-4a4f-86e4-96cd0a5b527e None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:45:00.722459 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:45:00.775429 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1bd10e-0d30-4a4f-86e4-96cd0a5b527e None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:45:00.812412 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a10f68-e9a1-4048-91f2-ea1189953bf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:45:00.813017 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a10f68-e9a1-4048-91f2-ea1189953bf4 None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:45:00.813294 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a10f68-e9a1-4048-91f2-ea1189953bf4 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:45:00.813553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a10f68-e9a1-4048-91f2-ea1189953bf4 None admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:45:00.814188 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a10f68-e9a1-4048-91f2-ea1189953bf4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:45:00.815632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a10f68-e9a1-4048-91f2-ea1189953bf4 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:45:00.839972 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 443/890] 10.222.0.16 () {60 vars in 1132 bytes} [Sun Oct 30 00:45:00 2022] POST /identity/v3/users => generated 311 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:45:02.986107 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632900f3-e0ad-4fe4-9192-109960fa9e69 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:45:02.986697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632900f3-e0ad-4fe4-9192-109960fa9e69 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:45:02.986697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632900f3-e0ad-4fe4-9192-109960fa9e69 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:45:02.988016 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 448/891] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:45:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:45:02.994388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7c045f-8c32-493c-9e63-9f21e08fe851 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:45:02.995085 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7c045f-8c32-493c-9e63-9f21e08fe851 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:45:02.995193 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7c045f-8c32-493c-9e63-9f21e08fe851 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:45:03.017858 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-bc7c045f-8c32-493c-9e63-9f21e08fe851 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:45:03.019044 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-bc7c045f-8c32-493c-9e63-9f21e08fe851 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:45:03.100038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 444/892] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:45:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:45:03.361847 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632900f3-e0ad-4fe4-9192-109960fa9e69 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:45:03.468935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632900f3-e0ad-4fe4-9192-109960fa9e69 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:45:03.505993 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8837bc0-ffb4-451c-91e8-5b02966f0111 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:45:03.506607 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8837bc0-ffb4-451c-91e8-5b02966f0111 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:45:03.506884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8837bc0-ffb4-451c-91e8-5b02966f0111 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:45:03.507152 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8837bc0-ffb4-451c-91e8-5b02966f0111 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:45:03.518083 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8837bc0-ffb4-451c-91e8-5b02966f0111 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:45:03.522809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-f8837bc0-ffb4-451c-91e8-5b02966f0111 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:45:03.522809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:45:03.522809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:45:03.522809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:45:03.522809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:45:03.522809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:45:03.522809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:45:03.522809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:45:03.522809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:45:03.522809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:45:03.522809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:45:03.522809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:45:03.522809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:45:03.522809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:45:03.522809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:45:03.522809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:45:03.523944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:45:03.523944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:45:03.523944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:45:03.523944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:45:03.523944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:45:03.523944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:45:03.523944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:45:03.523944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:45:03.523944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:45:03.523944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:45:03.523944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:45:03.523944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:45:03.523944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:45:03.523944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:45:03.523944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:45:03.523944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:45:03.525025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:45:03.525025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:45:03.525025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:45:03.525025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:45:03.525025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:45:03.525025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:45:03.525025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:45:03.525025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:45:03.525025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:45:03.525025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:45:03.525025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:45:03.525025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:45:03.525025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 449/893] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:45:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:45:03.530088 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7c045f-8c32-493c-9e63-9f21e08fe851 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:45:03.579945 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:45:03.644302 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7c045f-8c32-493c-9e63-9f21e08fe851 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:45:03.678821 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c92d00f-45f7-4b28-85ed-40778cc4097e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:45:03.679415 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c92d00f-45f7-4b28-85ed-40778cc4097e None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:45:03.679671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c92d00f-45f7-4b28-85ed-40778cc4097e None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:45:03.679975 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c92d00f-45f7-4b28-85ed-40778cc4097e None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:45:03.680633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c92d00f-45f7-4b28-85ed-40778cc4097e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:45:03.680926 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c92d00f-45f7-4b28-85ed-40778cc4097e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:45:03.682294 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c92d00f-45f7-4b28-85ed-40778cc4097e None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:45:03.690783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 445/894] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:45:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:45:03.700670 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8837bc0-ffb4-451c-91e8-5b02966f0111 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:45:03.725130 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:45:03.815556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8837bc0-ffb4-451c-91e8-5b02966f0111 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:45:03.853333 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c423b72-f10a-4443-a0d4-40ba83867a9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:45:03.853941 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c423b72-f10a-4443-a0d4-40ba83867a9f None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:45:03.854207 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c423b72-f10a-4443-a0d4-40ba83867a9f None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:45:03.854469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c423b72-f10a-4443-a0d4-40ba83867a9f None admin] PATH_INFO: `/v3/users/edc42a21a18048c4818777defaaa6aa7` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:45:03.869127 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c423b72-f10a-4443-a0d4-40ba83867a9f None admin] RBAC: Authorizing `identity:get_user(user_id=edc42a21a18048c4818777defaaa6aa7)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:45:03.870553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c423b72-f10a-4443-a0d4-40ba83867a9f None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:45:03.872140 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 450/895] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:45:03 2022] GET /identity/v3/users/edc42a21a18048c4818777defaaa6aa7 => generated 311 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:45:03.880984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c92d00f-45f7-4b28-85ed-40778cc4097e None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:45:03.994139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c92d00f-45f7-4b28-85ed-40778cc4097e None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:45:04.002621 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:45:04.031017 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2b616c-2b5b-4d10-8941-67985eb5de83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:45:04.031627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2b616c-2b5b-4d10-8941-67985eb5de83 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:45:04.031932 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2b616c-2b5b-4d10-8941-67985eb5de83 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:45:04.032197 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2b616c-2b5b-4d10-8941-67985eb5de83 None admin] PATH_INFO: `/v3/projects/a7dc1209bd974316b3d8f47c975a152d` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:45:04.033406 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2b616c-2b5b-4d10-8941-67985eb5de83 None admin] RBAC: Authorizing `identity:get_project(project_id=a7dc1209bd974316b3d8f47c975a152d)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:45:04.034823 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2b616c-2b5b-4d10-8941-67985eb5de83 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:45:04.036486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 446/896] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:45:03 2022] GET /identity/v3/projects/a7dc1209bd974316b3d8f47c975a152d => generated 361 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:45:04.042896 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c423b72-f10a-4443-a0d4-40ba83867a9f None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:45:04.113078 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:45:04.143376 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c423b72-f10a-4443-a0d4-40ba83867a9f None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:45:04.179336 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5151b4-b40e-4a7e-b812-4d55bfb5f07c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:45:04.180013 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5151b4-b40e-4a7e-b812-4d55bfb5f07c None admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:45:04.180280 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5151b4-b40e-4a7e-b812-4d55bfb5f07c None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:45:04.180539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5151b4-b40e-4a7e-b812-4d55bfb5f07c None admin] PATH_INFO: `/v3/projects/a7dc1209bd974316b3d8f47c975a152d/users/edc42a21a18048c4818777defaaa6aa7/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:45:04.182832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5151b4-b40e-4a7e-b812-4d55bfb5f07c None admin] RBAC: Authorizing `identity:create_grant(project_id=a7dc1209bd974316b3d8f47c975a152d, user_id=edc42a21a18048c4818777defaaa6aa7, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:45:04.184309 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5151b4-b40e-4a7e-b812-4d55bfb5f07c None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:45:04.202635 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 451/897] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:45:04 2022] PUT /identity/v3/projects/a7dc1209bd974316b3d8f47c975a152d/users/edc42a21a18048c4818777defaaa6aa7/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:45:06.299061 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58833f0-c070-4df0-8b9d-4ad10f2e8131 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:45:06.299966 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58833f0-c070-4df0-8b9d-4ad10f2e8131 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:45:06.299966 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58833f0-c070-4df0-8b9d-4ad10f2e8131 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:45:06.301688 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 447/898] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:45:06 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:45:06.308079 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33002a4b-75b3-425f-b48c-3fba737532ea None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:45:06.308512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33002a4b-75b3-425f-b48c-3fba737532ea None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:45:06.308628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33002a4b-75b3-425f-b48c-3fba737532ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:45:06.335812 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-33002a4b-75b3-425f-b48c-3fba737532ea None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:45:06.336989 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-33002a4b-75b3-425f-b48c-3fba737532ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:45:06.411109 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 452/899] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:45:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:45:06.681104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58833f0-c070-4df0-8b9d-4ad10f2e8131 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:45:06.796697 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58833f0-c070-4df0-8b9d-4ad10f2e8131 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:45:06.834946 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf9d50b-6bf4-4ce9-848e-fd5ed673fe69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:45:06.835550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf9d50b-6bf4-4ce9-848e-fd5ed673fe69 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:45:06.835838 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf9d50b-6bf4-4ce9-848e-fd5ed673fe69 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:45:06.836100 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf9d50b-6bf4-4ce9-848e-fd5ed673fe69 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:45:06.847336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf9d50b-6bf4-4ce9-848e-fd5ed673fe69 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:45:06.852050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-4cf9d50b-6bf4-4ce9-848e-fd5ed673fe69 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:45:06.852050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:45:06.852050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:45:06.852050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:45:06.852050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:45:06.852050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:45:06.852050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:45:06.852050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:45:06.852050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:45:06.852050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:45:06.852050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:45:06.852050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:45:06.852050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:45:06.852050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:45:06.852050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:45:06.852050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:45:06.853125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:45:06.853125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:45:06.853125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:45:06.853125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:45:06.853125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:45:06.853125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:45:06.853125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:45:06.853125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:45:06.853125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:45:06.853125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:45:06.853125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:45:06.853125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:45:06.853125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:45:06.853125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:45:06.853125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:45:06.853125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:45:06.854002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:45:06.854002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:45:06.854002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:45:06.854002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:45:06.854002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:45:06.854002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:45:06.854002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:45:06.854002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:45:06.854002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:45:06.854002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:45:06.854002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:45:06.854002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:45:06.854002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 448/900] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:45:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:45:06.861611 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33002a4b-75b3-425f-b48c-3fba737532ea None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:45:06.927325 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:45:06.969233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33002a4b-75b3-425f-b48c-3fba737532ea None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:45:07.004426 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458f6af8-09e3-4fca-b295-69c467384fe8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:45:07.005042 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458f6af8-09e3-4fca-b295-69c467384fe8 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:45:07.005274 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458f6af8-09e3-4fca-b295-69c467384fe8 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:45:07.005559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458f6af8-09e3-4fca-b295-69c467384fe8 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:45:07.006211 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458f6af8-09e3-4fca-b295-69c467384fe8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:45:07.006502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458f6af8-09e3-4fca-b295-69c467384fe8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:45:07.007899 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458f6af8-09e3-4fca-b295-69c467384fe8 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:45:07.015872 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 453/901] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:45:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:45:07.023493 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf9d50b-6bf4-4ce9-848e-fd5ed673fe69 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:45:07.059287 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:45:07.135166 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf9d50b-6bf4-4ce9-848e-fd5ed673fe69 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:45:07.168981 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94863154-4ede-4cb9-b38d-e9b010cc36c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:45:07.169590 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94863154-4ede-4cb9-b38d-e9b010cc36c2 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:45:07.169860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94863154-4ede-4cb9-b38d-e9b010cc36c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:45:07.170121 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94863154-4ede-4cb9-b38d-e9b010cc36c2 None admin] PATH_INFO: `/v3/users/edc42a21a18048c4818777defaaa6aa7` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:45:07.171246 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94863154-4ede-4cb9-b38d-e9b010cc36c2 None admin] RBAC: Authorizing `identity:get_user(user_id=edc42a21a18048c4818777defaaa6aa7)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:45:07.172689 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94863154-4ede-4cb9-b38d-e9b010cc36c2 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:45:07.174236 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 449/902] 10.222.0.16 () {56 vars in 1177 bytes} [Sun Oct 30 00:45:07 2022] GET /identity/v3/users/edc42a21a18048c4818777defaaa6aa7 => generated 311 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:45:07.182993 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458f6af8-09e3-4fca-b295-69c467384fe8 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:45:07.284839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458f6af8-09e3-4fca-b295-69c467384fe8 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:45:07.308039 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:45:07.324193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c52fbc-70d4-424b-a739-085951f8ea15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:45:07.324789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c52fbc-70d4-424b-a739-085951f8ea15 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:45:07.325048 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c52fbc-70d4-424b-a739-085951f8ea15 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:45:07.325315 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c52fbc-70d4-424b-a739-085951f8ea15 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:45:07.394528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c52fbc-70d4-424b-a739-085951f8ea15 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:45:07.397080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c52fbc-70d4-424b-a739-085951f8ea15 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:45:07.502165 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 454/903] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:45:07 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 30 00:45:07.508393 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94863154-4ede-4cb9-b38d-e9b010cc36c2 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:45:07.626012 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94863154-4ede-4cb9-b38d-e9b010cc36c2 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:45:07.628356 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:45:07.661841 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889ad33e-4d47-4ab2-833a-1cbc87ccb84f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:45:07.662480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889ad33e-4d47-4ab2-833a-1cbc87ccb84f None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:45:07.662752 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889ad33e-4d47-4ab2-833a-1cbc87ccb84f None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:45:07.663023 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889ad33e-4d47-4ab2-833a-1cbc87ccb84f None admin] PATH_INFO: `/v3/projects/a7dc1209bd974316b3d8f47c975a152d` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:45:07.664224 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889ad33e-4d47-4ab2-833a-1cbc87ccb84f None admin] RBAC: Authorizing `identity:get_project(project_id=a7dc1209bd974316b3d8f47c975a152d)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:45:07.665659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889ad33e-4d47-4ab2-833a-1cbc87ccb84f None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:45:07.667295 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 450/904] 10.222.0.16 () {56 vars in 1186 bytes} [Sun Oct 30 00:45:07 2022] GET /identity/v3/projects/a7dc1209bd974316b3d8f47c975a152d => generated 361 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:45:07.673873 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c52fbc-70d4-424b-a739-085951f8ea15 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:45:07.698976 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:45:07.773916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c52fbc-70d4-424b-a739-085951f8ea15 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:45:07.808680 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba5c6eb-70b7-4fb0-a203-6220b1e20d94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:45:07.809268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba5c6eb-70b7-4fb0-a203-6220b1e20d94 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:45:07.809539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba5c6eb-70b7-4fb0-a203-6220b1e20d94 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:45:07.809805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba5c6eb-70b7-4fb0-a203-6220b1e20d94 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:45:07.810514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba5c6eb-70b7-4fb0-a203-6220b1e20d94 None admin] RBAC: Adding query filter params (role.id=35f59686b5d84291a8d9baf19287eb5a, scope.project.id=a7dc1209bd974316b3d8f47c975a152d, user.id=edc42a21a18048c4818777defaaa6aa7) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:45:07.810813 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba5c6eb-70b7-4fb0-a203-6220b1e20d94 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:45:07.812594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba5c6eb-70b7-4fb0-a203-6220b1e20d94 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:45:07.818379 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 455/905] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:45:07 2022] GET /identity/v3/role_assignments?user.id=edc42a21a18048c4818777defaaa6aa7&scope.project.id=a7dc1209bd974316b3d8f47c975a152d&role.id=35f59686b5d84291a8d9baf19287eb5a => generated 608 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:45:17.656990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=14 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:45:17.815729 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=56 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:03.517290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074608a0-f9a1-4ee9-b647-8a7d6a4af419 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:03.517896 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074608a0-f9a1-4ee9-b647-8a7d6a4af419 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:03.517896 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074608a0-f9a1-4ee9-b647-8a7d6a4af419 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:03.519501 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 451/906] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:46:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:46:03.525374 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cb78f9-a2b5-49e7-81e4-bd909983b42b None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:03.525875 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cb78f9-a2b5-49e7-81e4-bd909983b42b None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:03.525875 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27cb78f9-a2b5-49e7-81e4-bd909983b42b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:03.545208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-27cb78f9-a2b5-49e7-81e4-bd909983b42b None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:46:03.546402 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-27cb78f9-a2b5-49e7-81e4-bd909983b42b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:46:03.626363 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 456/907] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:46:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 30 00:46:03.910074 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074608a0-f9a1-4ee9-b647-8a7d6a4af419 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:04.035633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074608a0-f9a1-4ee9-b647-8a7d6a4af419 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:04.077427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a155fc03-5f63-41ce-aa57-59ec2ece419f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:04.078086 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a155fc03-5f63-41ce-aa57-59ec2ece419f None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:04.078332 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a155fc03-5f63-41ce-aa57-59ec2ece419f None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:04.078600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a155fc03-5f63-41ce-aa57-59ec2ece419f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:04.091764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a155fc03-5f63-41ce-aa57-59ec2ece419f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:04.093248 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a155fc03-5f63-41ce-aa57-59ec2ece419f None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:04.114921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-a155fc03-5f63-41ce-aa57-59ec2ece419f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:46:04.114921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:04.114921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:04.114921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:04.114921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:04.114921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:04.114921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:04.114921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:04.114921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:04.114921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:04.114921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:04.114921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:04.114921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:46:04.114921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:46:04.114921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:46:04.114921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:46:04.116137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:04.116137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:04.116137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:04.116137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:04.116137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:04.116137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:04.116137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:04.116137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:04.116137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:04.116137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:04.116137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:04.116137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:04.116137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:04.116137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:04.116137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:04.116137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:04.116137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:46:04.117665 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:46:04.117665 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:46:04.117665 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:46:04.117665 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 452/908] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:46:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:46:04.126225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cb78f9-a2b5-49e7-81e4-bd909983b42b None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:04.256542 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27cb78f9-a2b5-49e7-81e4-bd909983b42b None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:04.293802 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:04.301381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7074c70-cf5c-4bf9-8f98-82971297e50d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:04.302067 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7074c70-cf5c-4bf9-8f98-82971297e50d None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:04.302394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7074c70-cf5c-4bf9-8f98-82971297e50d None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:04.302661 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7074c70-cf5c-4bf9-8f98-82971297e50d None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:04.303335 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7074c70-cf5c-4bf9-8f98-82971297e50d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:04.303663 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7074c70-cf5c-4bf9-8f98-82971297e50d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:04.305066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7074c70-cf5c-4bf9-8f98-82971297e50d None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:04.323378 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 457/909] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:46:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:04.325942 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a155fc03-5f63-41ce-aa57-59ec2ece419f None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:04.399854 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:04.453939 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a155fc03-5f63-41ce-aa57-59ec2ece419f None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:04.495244 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecce9f5-2550-4bb0-b9e1-a35f95558a49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:04.495907 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecce9f5-2550-4bb0-b9e1-a35f95558a49 None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:04.496184 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecce9f5-2550-4bb0-b9e1-a35f95558a49 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:04.496454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecce9f5-2550-4bb0-b9e1-a35f95558a49 None admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:04.497118 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecce9f5-2550-4bb0-b9e1-a35f95558a49 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:04.498567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecce9f5-2550-4bb0-b9e1-a35f95558a49 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:04.527898 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 453/910] 10.222.0.16 () {60 vars in 1132 bytes} [Sun Oct 30 00:46:04 2022] POST /identity/v3/users => generated 251 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:46:06.893590 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a887c9-51d6-4821-9539-ac430f741296 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:06.894176 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a887c9-51d6-4821-9539-ac430f741296 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:06.894176 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a887c9-51d6-4821-9539-ac430f741296 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:06.895495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 458/911] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:46:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:46:06.902832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f19d57f-6a5b-4327-8070-72130bf5f9f4 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:06.903371 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f19d57f-6a5b-4327-8070-72130bf5f9f4 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:06.903624 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f19d57f-6a5b-4327-8070-72130bf5f9f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:06.926679 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-7f19d57f-6a5b-4327-8070-72130bf5f9f4 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:46:06.928334 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-7f19d57f-6a5b-4327-8070-72130bf5f9f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:46:07.008042 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 454/912] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:46:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 30 00:46:07.300437 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a887c9-51d6-4821-9539-ac430f741296 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:07.425284 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a887c9-51d6-4821-9539-ac430f741296 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:07.462298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb775e9-6de6-4b9f-b3ae-84e3129bfbb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:07.463160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb775e9-6de6-4b9f-b3ae-84e3129bfbb5 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:07.463450 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb775e9-6de6-4b9f-b3ae-84e3129bfbb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:07.463753 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb775e9-6de6-4b9f-b3ae-84e3129bfbb5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:07.475342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb775e9-6de6-4b9f-b3ae-84e3129bfbb5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:46:07.480486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-9cb775e9-6de6-4b9f-b3ae-84e3129bfbb5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:46:07.480486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:07.480486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:07.480486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:07.480486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:07.480486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:07.480486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:07.480486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:07.480486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:07.480486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:07.480486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:07.480486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:07.480486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:46:07.480486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:46:07.480486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:46:07.480486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:46:07.481586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:46:07.481586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:46:07.481586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:46:07.481586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:46:07.481586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:07.481586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:07.481586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:07.481586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:07.481586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:07.481586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:07.481586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:07.481586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:07.481586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:07.481586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:07.481586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:07.481586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:07.482697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:07.482697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:07.482697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:07.482697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:07.482697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:46:07.482697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:46:07.482697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:46:07.482697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:46:07.482697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:46:07.482697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:46:07.482697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:46:07.482697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:46:07.482697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 459/913] 10.222.0.16 () {56 vars in 1102 bytes} [Sun Oct 30 00:46:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:46:07.488281 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f19d57f-6a5b-4327-8070-72130bf5f9f4 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:07.534172 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:07.606574 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f19d57f-6a5b-4327-8070-72130bf5f9f4 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:07.650182 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25bad9a3-e8df-49dc-bdca-49a15a19aafd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:07.651069 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25bad9a3-e8df-49dc-bdca-49a15a19aafd None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:07.651367 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25bad9a3-e8df-49dc-bdca-49a15a19aafd None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:07.651691 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25bad9a3-e8df-49dc-bdca-49a15a19aafd None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:07.652559 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25bad9a3-e8df-49dc-bdca-49a15a19aafd None admin] RBAC: Adding query filter params (name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:07.652913 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25bad9a3-e8df-49dc-bdca-49a15a19aafd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:07.654523 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25bad9a3-e8df-49dc-bdca-49a15a19aafd None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:07.672065 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 455/914] 10.222.0.16 () {56 vars in 1103 bytes} [Sun Oct 30 00:46:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:07.676723 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb775e9-6de6-4b9f-b3ae-84e3129bfbb5 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:07.737757 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:07.805441 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb775e9-6de6-4b9f-b3ae-84e3129bfbb5 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:07.848794 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8439af-a547-4c89-b603-58bf660764b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:07.852414 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8439af-a547-4c89-b603-58bf660764b8 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:07.852737 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8439af-a547-4c89-b603-58bf660764b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:07.853010 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8439af-a547-4c89-b603-58bf660764b8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:07.865162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8439af-a547-4c89-b603-58bf660764b8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:07.866929 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8439af-a547-4c89-b603-58bf660764b8 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:07.883089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-5c8439af-a547-4c89-b603-58bf660764b8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:46:07.883089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:07.883089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:07.883089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:07.883089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:07.883089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:07.883089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:07.883089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:07.883089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:07.883089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:07.883089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:07.883089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:07.883089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:46:07.883089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:46:07.883089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:46:07.883089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:46:07.889648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:07.889648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:07.889648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:07.889648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:07.889648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:07.889648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:07.889648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:07.889648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:07.889648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:07.889648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:07.889648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:07.889648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:07.889648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:07.889648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:07.889648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:07.889648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:07.889648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:46:07.890794 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:46:07.890794 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:46:07.890794 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:46:07.890794 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 460/915] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:46:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:46:07.891082 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25bad9a3-e8df-49dc-bdca-49a15a19aafd None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:08.004365 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:08.007886 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25bad9a3-e8df-49dc-bdca-49a15a19aafd None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:08.053758 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d79719f-43e2-4522-889c-5e63fb66efa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:08.054721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d79719f-43e2-4522-889c-5e63fb66efa4 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:08.054990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d79719f-43e2-4522-889c-5e63fb66efa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:08.055258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d79719f-43e2-4522-889c-5e63fb66efa4 None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:08.055930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d79719f-43e2-4522-889c-5e63fb66efa4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:08.056226 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d79719f-43e2-4522-889c-5e63fb66efa4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:08.057580 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d79719f-43e2-4522-889c-5e63fb66efa4 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:08.072662 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 456/916] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:46:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:46:08.079037 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8439af-a547-4c89-b603-58bf660764b8 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:08.206167 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8439af-a547-4c89-b603-58bf660764b8 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:08.207922 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:08.245663 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc60d66-1868-4e9f-a859-e0e7ff6f8c80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:08.246315 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc60d66-1868-4e9f-a859-e0e7ff6f8c80 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:08.246591 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc60d66-1868-4e9f-a859-e0e7ff6f8c80 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:08.246859 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc60d66-1868-4e9f-a859-e0e7ff6f8c80 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:08.256970 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc60d66-1868-4e9f-a859-e0e7ff6f8c80 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:08.258690 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc60d66-1868-4e9f-a859-e0e7ff6f8c80 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:08.272559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-2dc60d66-1868-4e9f-a859-e0e7ff6f8c80 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 30 00:46:08.272559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:08.272559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:08.272559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:08.272559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:08.272559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:08.272559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:08.272559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:08.272559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:08.272559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:08.272559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:08.272559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:08.272559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:46:08.272559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:46:08.272559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:46:08.272559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:46:08.278528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:08.278528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:08.278528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:46:08.278528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:46:08.278528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:46:08.278528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:46:08.278528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:08.278528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:08.278528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:08.278528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:08.278528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:08.278528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:08.278528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:08.278528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:08.278528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:08.278528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:08.278528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:08.278528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:08.279693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:08.279693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:08.279693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:46:08.279693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:46:08.279693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:46:08.279693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:46:08.279693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:46:08.279693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:46:08.279693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 30 00:46:08.279693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:46:08.279693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 461/917] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:46:08 2022] GET /identity/v3/users/placement => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:46:08.281747 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d79719f-43e2-4522-889c-5e63fb66efa4 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:08.401282 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d79719f-43e2-4522-889c-5e63fb66efa4 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:08.439669 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba9828e-f44b-4d18-9e62-818a68b07283 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:08.440305 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba9828e-f44b-4d18-9e62-818a68b07283 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:08.440620 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba9828e-f44b-4d18-9e62-818a68b07283 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:08.440977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba9828e-f44b-4d18-9e62-818a68b07283 None admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:08.441787 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba9828e-f44b-4d18-9e62-818a68b07283 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:08.442076 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba9828e-f44b-4d18-9e62-818a68b07283 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:08.443956 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba9828e-f44b-4d18-9e62-818a68b07283 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:08.463856 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 457/918] 10.222.0.16 () {56 vars in 1143 bytes} [Sun Oct 30 00:46:08 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:08.469433 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc60d66-1868-4e9f-a859-e0e7ff6f8c80 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:08.579874 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc60d66-1868-4e9f-a859-e0e7ff6f8c80 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:08.618524 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca53321-6b60-45f0-8f10-b7ce65dc3905 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:08.619156 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca53321-6b60-45f0-8f10-b7ce65dc3905 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:08.619427 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca53321-6b60-45f0-8f10-b7ce65dc3905 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:08.619687 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca53321-6b60-45f0-8f10-b7ce65dc3905 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:08.630484 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca53321-6b60-45f0-8f10-b7ce65dc3905 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:08.631993 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca53321-6b60-45f0-8f10-b7ce65dc3905 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:08.646483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-dca53321-6b60-45f0-8f10-b7ce65dc3905 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:46:08.646483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:08.646483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:08.646483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:08.646483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:08.646483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:08.646483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:08.646483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:08.646483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:08.646483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:08.646483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:08.646483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:08.646483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:46:08.646483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:46:08.646483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:46:08.646483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:46:08.647520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:08.647520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:08.647520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:08.647520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:08.647520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:08.647520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:08.647520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:08.647520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:08.647520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:08.647520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:08.647520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:08.647520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:08.647520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:08.647520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:08.647520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:08.647520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:08.647520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:46:08.648449 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:46:08.648449 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:46:08.648449 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:46:08.654084 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 462/919] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:46:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:46:08.654230 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba9828e-f44b-4d18-9e62-818a68b07283 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:08.673466 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:08.767548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba9828e-f44b-4d18-9e62-818a68b07283 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:08.804589 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef06a5c8-f8fe-44a0-8e78-4d306c5228fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:08.805337 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef06a5c8-f8fe-44a0-8e78-4d306c5228fb None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:08.805711 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef06a5c8-f8fe-44a0-8e78-4d306c5228fb None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:08.806076 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef06a5c8-f8fe-44a0-8e78-4d306c5228fb None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:08.806836 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef06a5c8-f8fe-44a0-8e78-4d306c5228fb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:08.807231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef06a5c8-f8fe-44a0-8e78-4d306c5228fb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:08.808734 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef06a5c8-f8fe-44a0-8e78-4d306c5228fb None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:08.825952 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 458/920] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:46:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:08.827449 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca53321-6b60-45f0-8f10-b7ce65dc3905 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:08.847770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:08.952227 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca53321-6b60-45f0-8f10-b7ce65dc3905 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:08.991347 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0995b2ad-7512-4f41-a1bf-ee718ad9f91f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:08.992030 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0995b2ad-7512-4f41-a1bf-ee718ad9f91f None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:08.992306 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0995b2ad-7512-4f41-a1bf-ee718ad9f91f None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:08.992573 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0995b2ad-7512-4f41-a1bf-ee718ad9f91f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:09.000796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0995b2ad-7512-4f41-a1bf-ee718ad9f91f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:09.002436 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0995b2ad-7512-4f41-a1bf-ee718ad9f91f None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:09.015149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-0995b2ad-7512-4f41-a1bf-ee718ad9f91f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:46:09.015149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:09.015149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:09.015149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:09.015149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:09.015149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:09.015149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:09.015149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:09.015149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:09.015149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:09.015149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:09.015149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:09.015149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:46:09.015149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:46:09.015149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:46:09.016147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:46:09.016147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:09.016147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:09.016147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:09.016147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:09.016147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:09.016147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:09.016147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:09.016147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:09.016147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:09.016147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:09.016147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:09.016147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:09.016147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:09.016147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:09.016147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:09.016147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:09.017041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:46:09.017041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:46:09.017041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:46:09.017041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:46:09.017041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:46:09.017041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:46:09.017041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:46:09.017041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:46:09.017041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 463/921] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:46:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:46:09.022564 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef06a5c8-f8fe-44a0-8e78-4d306c5228fb None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:09.119487 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:09.144131 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef06a5c8-f8fe-44a0-8e78-4d306c5228fb None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:09.183373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe82ad9a-a790-4cd1-a72c-941d21441a36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:09.184032 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe82ad9a-a790-4cd1-a72c-941d21441a36 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:09.184320 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe82ad9a-a790-4cd1-a72c-941d21441a36 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:09.184564 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe82ad9a-a790-4cd1-a72c-941d21441a36 None admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:09.185480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe82ad9a-a790-4cd1-a72c-941d21441a36 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:09.185783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe82ad9a-a790-4cd1-a72c-941d21441a36 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:09.187235 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe82ad9a-a790-4cd1-a72c-941d21441a36 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:09.203913 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 459/922] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:46:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:09.206368 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0995b2ad-7512-4f41-a1bf-ee718ad9f91f None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:09.292468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:09.325868 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0995b2ad-7512-4f41-a1bf-ee718ad9f91f None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:09.366100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d427ed-369b-4850-92d2-312675a4f391 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:09.367063 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d427ed-369b-4850-92d2-312675a4f391 None admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:09.367542 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d427ed-369b-4850-92d2-312675a4f391 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:09.367809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d427ed-369b-4850-92d2-312675a4f391 None admin] PATH_INFO: `/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/63cc035057064805a95e9f12714ebade/roles/5a0f6f642a10411da6fc1bfc92a7a3d3` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:09.385007 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d427ed-369b-4850-92d2-312675a4f391 None admin] RBAC: Authorizing `identity:create_grant(project_id=658799ed31d2472fb36af4aff0f03aab, user_id=63cc035057064805a95e9f12714ebade, role_id=5a0f6f642a10411da6fc1bfc92a7a3d3)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:09.388867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d427ed-369b-4850-92d2-312675a4f391 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:09.401856 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 464/923] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:46:09 2022] PUT /identity/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/63cc035057064805a95e9f12714ebade/roles/5a0f6f642a10411da6fc1bfc92a7a3d3 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 00:46:11.565071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ee4e59-6e67-4c35-94b2-683d5fda4788 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:11.565652 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ee4e59-6e67-4c35-94b2-683d5fda4788 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:11.565652 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ee4e59-6e67-4c35-94b2-683d5fda4788 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:11.566949 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 460/924] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:46:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:46:11.572491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dce452-c3c7-4e85-9e14-d74ee92b45c2 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:11.572915 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dce452-c3c7-4e85-9e14-d74ee92b45c2 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:11.573026 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dce452-c3c7-4e85-9e14-d74ee92b45c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:11.591776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-34dce452-c3c7-4e85-9e14-d74ee92b45c2 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:46:11.593014 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-34dce452-c3c7-4e85-9e14-d74ee92b45c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:46:11.661518 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 465/925] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:46:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:46:11.924120 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ee4e59-6e67-4c35-94b2-683d5fda4788 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:12.034372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ee4e59-6e67-4c35-94b2-683d5fda4788 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:12.076106 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fd8d23-3233-43b3-93ec-cdd5d6d7d189 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:12.077163 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fd8d23-3233-43b3-93ec-cdd5d6d7d189 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:12.077570 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fd8d23-3233-43b3-93ec-cdd5d6d7d189 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:12.077961 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fd8d23-3233-43b3-93ec-cdd5d6d7d189 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:12.087162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:12.091737 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92fd8d23-3233-43b3-93ec-cdd5d6d7d189 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:46:12.097201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-92fd8d23-3233-43b3-93ec-cdd5d6d7d189 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:46:12.097201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:12.097201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:12.097201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:12.097201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:12.097201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:12.097201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:12.097201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:12.097201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:12.097201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:12.097201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:12.097201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:12.097201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:46:12.097201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:46:12.097201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:46:12.097201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:46:12.098373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:46:12.098373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:46:12.098373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:46:12.098373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:46:12.098373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:12.098373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:12.098373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:12.098373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:12.098373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:12.098373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:12.098373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:12.098373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:12.098373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:12.098373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:12.098373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:12.098373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:12.099366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:12.099366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:12.099366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:12.099366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:12.099366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:46:12.099366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:46:12.099366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:46:12.099366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:46:12.099366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:46:12.099366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:46:12.099366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 30 00:46:12.099366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:46:12.099366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 461/926] 10.222.0.16 () {56 vars in 1102 bytes} [Sun Oct 30 00:46:11 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:46:12.105085 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dce452-c3c7-4e85-9e14-d74ee92b45c2 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:12.162109 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:12.210791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dce452-c3c7-4e85-9e14-d74ee92b45c2 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:12.248244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4200b6c1-6d01-4948-989c-d8fa4732fe45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:12.248856 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4200b6c1-6d01-4948-989c-d8fa4732fe45 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:12.249138 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4200b6c1-6d01-4948-989c-d8fa4732fe45 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:12.249451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4200b6c1-6d01-4948-989c-d8fa4732fe45 None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:12.250261 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4200b6c1-6d01-4948-989c-d8fa4732fe45 None admin] RBAC: Adding query filter params (name=service) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:12.250578 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4200b6c1-6d01-4948-989c-d8fa4732fe45 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:12.252211 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4200b6c1-6d01-4948-989c-d8fa4732fe45 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:12.261904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 466/927] 10.222.0.16 () {56 vars in 1103 bytes} [Sun Oct 30 00:46:12 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:12.268161 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fd8d23-3233-43b3-93ec-cdd5d6d7d189 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:12.394608 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fd8d23-3233-43b3-93ec-cdd5d6d7d189 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:12.432039 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72714909-0bc6-4f4b-bcd6-74eaec700cb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:12.432731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72714909-0bc6-4f4b-bcd6-74eaec700cb0 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:12.433027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72714909-0bc6-4f4b-bcd6-74eaec700cb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:12.433312 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72714909-0bc6-4f4b-bcd6-74eaec700cb0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:12.445127 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72714909-0bc6-4f4b-bcd6-74eaec700cb0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:12.446951 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72714909-0bc6-4f4b-bcd6-74eaec700cb0 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:12.466443 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-72714909-0bc6-4f4b-bcd6-74eaec700cb0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:46:12.466443 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:12.466443 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:12.466443 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:12.466443 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:12.466443 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:12.466443 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:12.466443 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:12.466443 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:12.466443 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:12.466443 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:12.466443 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:12.466443 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:46:12.466443 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:46:12.466443 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:46:12.466443 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:46:12.467520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:12.467520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:12.467520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:12.467520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:12.467520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:12.467520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:12.467520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:12.467520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:12.467520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:12.467520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:12.467520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:12.467520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:12.467520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:12.467520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:12.467520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:12.467520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:12.467520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:46:12.473542 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:46:12.473542 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:46:12.473542 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:46:12.473542 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 462/928] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:46:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:46:12.475106 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4200b6c1-6d01-4948-989c-d8fa4732fe45 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:12.596765 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4200b6c1-6d01-4948-989c-d8fa4732fe45 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:12.614591 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:12.634174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b19c29-4361-46c7-a719-4379122f73d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:12.634867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b19c29-4361-46c7-a719-4379122f73d8 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:12.635204 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b19c29-4361-46c7-a719-4379122f73d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:12.635543 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b19c29-4361-46c7-a719-4379122f73d8 None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:12.636280 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b19c29-4361-46c7-a719-4379122f73d8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:12.636605 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b19c29-4361-46c7-a719-4379122f73d8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:12.637949 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b19c29-4361-46c7-a719-4379122f73d8 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:12.650562 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 467/929] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:46:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:12.656590 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72714909-0bc6-4f4b-bcd6-74eaec700cb0 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:12.709410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:12.764188 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72714909-0bc6-4f4b-bcd6-74eaec700cb0 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:12.802433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0200ea-a507-4f46-aa29-dac50b03d0f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:12.803337 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0200ea-a507-4f46-aa29-dac50b03d0f0 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:12.803611 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0200ea-a507-4f46-aa29-dac50b03d0f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:12.803910 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0200ea-a507-4f46-aa29-dac50b03d0f0 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:12.812390 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0200ea-a507-4f46-aa29-dac50b03d0f0 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:12.813953 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0200ea-a507-4f46-aa29-dac50b03d0f0 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:12.826550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-db0200ea-a507-4f46-aa29-dac50b03d0f0 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 30 00:46:12.826550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:12.826550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:12.826550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:12.826550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:12.826550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:12.826550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:12.826550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:12.826550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:12.826550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:12.826550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:12.826550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:12.826550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:46:12.826550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:46:12.826550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:46:12.826550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:46:12.827731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:12.827731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:12.827731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:46:12.827731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:46:12.827731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:46:12.827731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:46:12.827731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:12.827731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:12.827731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:12.827731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:12.827731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:12.827731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:12.827731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:12.827731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:12.827731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:12.827731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:12.827731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:12.827731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:12.828682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:12.828682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:12.828682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:46:12.828682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:46:12.828682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:46:12.828682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:46:12.828682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:46:12.828682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:46:12.828682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 30 00:46:12.828682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:46:12.832427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 463/930] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:46:12 2022] GET /identity/v3/users/placement => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:46:12.834121 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b19c29-4361-46c7-a719-4379122f73d8 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:12.948680 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b19c29-4361-46c7-a719-4379122f73d8 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:12.987588 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3375182-58c4-4a14-9398-920c6851f667 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:12.988208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3375182-58c4-4a14-9398-920c6851f667 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:12.988490 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3375182-58c4-4a14-9398-920c6851f667 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:12.988770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3375182-58c4-4a14-9398-920c6851f667 None admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:12.989635 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3375182-58c4-4a14-9398-920c6851f667 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:12.989937 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3375182-58c4-4a14-9398-920c6851f667 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:12.991691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3375182-58c4-4a14-9398-920c6851f667 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:13.012052 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 468/931] 10.222.0.16 () {56 vars in 1143 bytes} [Sun Oct 30 00:46:12 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:13.020228 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0200ea-a507-4f46-aa29-dac50b03d0f0 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:13.140333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0200ea-a507-4f46-aa29-dac50b03d0f0 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:13.152391 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:13.178812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a6254f-2f05-447a-9410-6a6849b417c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:13.179410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a6254f-2f05-447a-9410-6a6849b417c2 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:13.179679 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a6254f-2f05-447a-9410-6a6849b417c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:13.179972 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a6254f-2f05-447a-9410-6a6849b417c2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:13.252865 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a6254f-2f05-447a-9410-6a6849b417c2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:13.254320 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a6254f-2f05-447a-9410-6a6849b417c2 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:13.355172 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 464/932] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:46:13 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 30 00:46:13.361223 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3375182-58c4-4a14-9398-920c6851f667 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:13.402009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:13.474112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3375182-58c4-4a14-9398-920c6851f667 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:13.512837 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deced533-ac42-454c-bcaa-60d9867eb1d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:13.513460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deced533-ac42-454c-bcaa-60d9867eb1d8 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:13.513751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deced533-ac42-454c-bcaa-60d9867eb1d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:13.514183 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deced533-ac42-454c-bcaa-60d9867eb1d8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:13.526038 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deced533-ac42-454c-bcaa-60d9867eb1d8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:13.527614 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deced533-ac42-454c-bcaa-60d9867eb1d8 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:13.546530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-deced533-ac42-454c-bcaa-60d9867eb1d8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:46:13.546530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:13.546530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:13.546530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:13.546530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:13.546530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:13.546530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:13.546530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:13.546530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:13.546530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:13.546530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:13.546530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:13.546530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:46:13.546530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:46:13.546530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:46:13.546530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:46:13.547579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:13.547579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:13.547579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:13.547579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:13.547579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:13.547579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:13.547579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:13.547579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:13.547579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:13.547579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:13.547579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:13.547579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:13.547579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:13.547579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:13.547579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:13.547579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:13.547579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:46:13.548510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:46:13.548510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:46:13.548510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:46:13.548510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 469/933] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:46:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:46:13.554328 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a6254f-2f05-447a-9410-6a6849b417c2 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:13.568755 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:13.668714 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a6254f-2f05-447a-9410-6a6849b417c2 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:13.707186 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76956624-c5dc-40e7-9113-d2ab31501d84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:13.707817 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76956624-c5dc-40e7-9113-d2ab31501d84 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:13.708105 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76956624-c5dc-40e7-9113-d2ab31501d84 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:13.708464 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76956624-c5dc-40e7-9113-d2ab31501d84 None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:13.709302 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76956624-c5dc-40e7-9113-d2ab31501d84 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:13.709663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76956624-c5dc-40e7-9113-d2ab31501d84 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:13.711133 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76956624-c5dc-40e7-9113-d2ab31501d84 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:13.724452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 465/934] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:46:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:13.730726 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deced533-ac42-454c-bcaa-60d9867eb1d8 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:13.823112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:13.838528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deced533-ac42-454c-bcaa-60d9867eb1d8 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:13.873560 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee841c29-9cfc-4b4b-b329-c75773a8607f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:13.874337 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee841c29-9cfc-4b4b-b329-c75773a8607f None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:13.874635 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee841c29-9cfc-4b4b-b329-c75773a8607f None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:13.874897 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee841c29-9cfc-4b4b-b329-c75773a8607f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:13.882101 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee841c29-9cfc-4b4b-b329-c75773a8607f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:13.883878 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee841c29-9cfc-4b4b-b329-c75773a8607f None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:13.896351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-ee841c29-9cfc-4b4b-b329-c75773a8607f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:46:13.896351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:13.896351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:13.896351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:13.896351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:13.896351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:13.896351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:13.896351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:13.896351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:13.896351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:13.896351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:13.896351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:13.896351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:46:13.896351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:46:13.896351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:46:13.902397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:46:13.902397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:13.902397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:13.902397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:13.902397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:13.902397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:13.902397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:13.902397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:13.902397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:13.902397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:13.902397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:13.902397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:13.902397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:13.902397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:13.902397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:13.902397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:13.902397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:13.903481 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:46:13.903481 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:46:13.903481 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:46:13.903481 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:46:13.903481 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:46:13.903481 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:46:13.903481 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:46:13.903481 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:46:13.903481 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 470/935] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:46:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:46:13.905044 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76956624-c5dc-40e7-9113-d2ab31501d84 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:13.980103 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:14.011834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76956624-c5dc-40e7-9113-d2ab31501d84 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:14.047047 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a59895-30c9-4c60-ae4b-395d1b9d85c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:14.047699 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a59895-30c9-4c60-ae4b-395d1b9d85c5 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:14.048012 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a59895-30c9-4c60-ae4b-395d1b9d85c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:14.048373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a59895-30c9-4c60-ae4b-395d1b9d85c5 None admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:14.048993 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a59895-30c9-4c60-ae4b-395d1b9d85c5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:14.049275 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a59895-30c9-4c60-ae4b-395d1b9d85c5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:14.050799 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a59895-30c9-4c60-ae4b-395d1b9d85c5 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:14.065209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 466/936] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:46:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:14.073584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee841c29-9cfc-4b4b-b329-c75773a8607f None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:14.190706 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee841c29-9cfc-4b4b-b329-c75773a8607f None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:14.226609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc450a59-fc72-459a-b176-9e32b23e7335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:14.227210 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc450a59-fc72-459a-b176-9e32b23e7335 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:14.227463 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc450a59-fc72-459a-b176-9e32b23e7335 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:14.227737 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc450a59-fc72-459a-b176-9e32b23e7335 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:14.228462 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc450a59-fc72-459a-b176-9e32b23e7335 None admin] RBAC: Adding query filter params (role.id=5a0f6f642a10411da6fc1bfc92a7a3d3, scope.project.id=658799ed31d2472fb36af4aff0f03aab, user.id=63cc035057064805a95e9f12714ebade) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:14.228755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc450a59-fc72-459a-b176-9e32b23e7335 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:14.230486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc450a59-fc72-459a-b176-9e32b23e7335 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:14.236569 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 471/937] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:46:14 2022] GET /identity/v3/role_assignments?user.id=63cc035057064805a95e9f12714ebade&scope.project.id=658799ed31d2472fb36af4aff0f03aab&role.id=5a0f6f642a10411da6fc1bfc92a7a3d3 => generated 608 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:46:16.414201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5de1c6-a75d-4662-8833-93889f97394a None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:16.414823 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5de1c6-a75d-4662-8833-93889f97394a None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:16.414823 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5de1c6-a75d-4662-8833-93889f97394a None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:16.416156 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 467/938] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:46:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:46:16.425829 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3663bd87-0550-4f9d-ace5-daca7597aa39 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:16.426290 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3663bd87-0550-4f9d-ace5-daca7597aa39 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:16.426386 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3663bd87-0550-4f9d-ace5-daca7597aa39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:16.446579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-3663bd87-0550-4f9d-ace5-daca7597aa39 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:46:16.447778 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3663bd87-0550-4f9d-ace5-daca7597aa39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:46:16.451666 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:16.522131 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 472/939] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:46:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:46:16.783157 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5de1c6-a75d-4662-8833-93889f97394a None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:16.898808 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5de1c6-a75d-4662-8833-93889f97394a None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:16.930394 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:16.936947 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c13daa1-3d7f-4d96-bae6-1e7f511e69c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:16.937644 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c13daa1-3d7f-4d96-bae6-1e7f511e69c6 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:16.937927 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c13daa1-3d7f-4d96-bae6-1e7f511e69c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:16.938214 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c13daa1-3d7f-4d96-bae6-1e7f511e69c6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:16.950060 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c13daa1-3d7f-4d96-bae6-1e7f511e69c6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:46:16.954847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-7c13daa1-3d7f-4d96-bae6-1e7f511e69c6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:46:16.954847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:16.954847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:16.954847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:16.954847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:16.954847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:16.954847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:16.954847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:16.954847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:16.954847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:16.954847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:16.954847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:16.954847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:46:16.954847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:46:16.954847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:46:16.954847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:46:16.956180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:46:16.956180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:46:16.956180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:46:16.956180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:46:16.956180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:16.956180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:16.956180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:16.956180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:16.956180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:16.956180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:16.956180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:16.956180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:16.956180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:16.956180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:16.956180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:16.956180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:16.957257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:16.957257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:16.957257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:16.957257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:16.957257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:46:16.957257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:46:16.957257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:46:16.957257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:46:16.957257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:46:16.957257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:46:16.957257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:46:16.957257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:46:16.957257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 468/940] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:46:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:46:16.962738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3663bd87-0550-4f9d-ace5-daca7597aa39 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:17.070700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3663bd87-0550-4f9d-ace5-daca7597aa39 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:17.107771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0531ff-ab30-4380-a44d-88d211cadd5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:17.108381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0531ff-ab30-4380-a44d-88d211cadd5f None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:17.108654 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0531ff-ab30-4380-a44d-88d211cadd5f None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:17.108918 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0531ff-ab30-4380-a44d-88d211cadd5f None admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:17.109624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0531ff-ab30-4380-a44d-88d211cadd5f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:17.109950 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0531ff-ab30-4380-a44d-88d211cadd5f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:17.111387 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0531ff-ab30-4380-a44d-88d211cadd5f None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:17.119654 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 473/941] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:46:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:17.125709 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c13daa1-3d7f-4d96-bae6-1e7f511e69c6 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:17.230020 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c13daa1-3d7f-4d96-bae6-1e7f511e69c6 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:17.267384 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee36e3d-5fa3-42a0-8b03-8c8342165011 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:17.268058 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee36e3d-5fa3-42a0-8b03-8c8342165011 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:17.268322 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee36e3d-5fa3-42a0-8b03-8c8342165011 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:17.268662 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee36e3d-5fa3-42a0-8b03-8c8342165011 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:17.280321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee36e3d-5fa3-42a0-8b03-8c8342165011 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:17.281805 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee36e3d-5fa3-42a0-8b03-8c8342165011 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:17.296467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-cee36e3d-5fa3-42a0-8b03-8c8342165011 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:46:17.296467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:17.296467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:17.296467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:17.296467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:17.296467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:17.296467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:17.296467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:17.296467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:17.296467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:17.296467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:17.296467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:17.296467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:46:17.296467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:46:17.296467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:46:17.296467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:46:17.297658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:17.297658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:17.297658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:17.297658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:17.297658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:17.297658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:17.297658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:17.297658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:17.297658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:17.297658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:17.297658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:17.297658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:17.297658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:17.297658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:17.297658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:17.297658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:17.297658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:46:17.298558 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:46:17.298558 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:46:17.298558 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:46:17.298558 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 469/942] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:46:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:46:17.303838 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0531ff-ab30-4380-a44d-88d211cadd5f None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:17.306780 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:17.410168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0531ff-ab30-4380-a44d-88d211cadd5f None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:17.445857 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1459a6-3c40-4d8e-a524-62ee2dedd366 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:17.446629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1459a6-3c40-4d8e-a524-62ee2dedd366 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:17.446773 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1459a6-3c40-4d8e-a524-62ee2dedd366 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:17.446987 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1459a6-3c40-4d8e-a524-62ee2dedd366 None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:17.447623 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1459a6-3c40-4d8e-a524-62ee2dedd366 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:17.447958 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1459a6-3c40-4d8e-a524-62ee2dedd366 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:17.449372 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1459a6-3c40-4d8e-a524-62ee2dedd366 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:17.462546 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 474/943] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:46:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:46:17.469406 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee36e3d-5fa3-42a0-8b03-8c8342165011 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:17.516677 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:17.575712 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee36e3d-5fa3-42a0-8b03-8c8342165011 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:17.610478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5602d3e1-7b38-405b-859e-029e637b55d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:17.611032 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5602d3e1-7b38-405b-859e-029e637b55d6 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:17.611302 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5602d3e1-7b38-405b-859e-029e637b55d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:17.611567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5602d3e1-7b38-405b-859e-029e637b55d6 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:17.619605 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5602d3e1-7b38-405b-859e-029e637b55d6 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:17.621073 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5602d3e1-7b38-405b-859e-029e637b55d6 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:17.634178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-5602d3e1-7b38-405b-859e-029e637b55d6 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 30 00:46:17.634178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:17.634178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:17.634178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:17.634178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:17.634178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:17.634178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:17.634178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:17.634178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:17.634178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:17.634178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:17.634178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:17.634178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:46:17.634178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:46:17.634178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:46:17.634178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:46:17.635381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:17.635381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:17.635381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:46:17.635381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:46:17.635381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:46:17.635381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:46:17.635381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:17.635381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:17.635381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:17.635381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:17.635381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:17.635381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:17.635381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:17.635381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:17.635381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:17.635381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:17.635381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:17.635381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:17.636392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:17.636392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:17.636392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:46:17.636392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:46:17.636392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:46:17.636392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:46:17.636392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:46:17.636392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:46:17.636392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 30 00:46:17.636392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:46:17.636392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 470/944] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:46:17 2022] GET /identity/v3/users/placement => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:46:17.641986 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1459a6-3c40-4d8e-a524-62ee2dedd366 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:17.709950 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:17.749331 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1459a6-3c40-4d8e-a524-62ee2dedd366 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:17.785897 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da566a16-a8a1-4ef0-9f6b-dffa570a1185 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:17.786502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da566a16-a8a1-4ef0-9f6b-dffa570a1185 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:17.786812 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da566a16-a8a1-4ef0-9f6b-dffa570a1185 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:17.787079 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da566a16-a8a1-4ef0-9f6b-dffa570a1185 None admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:17.787862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da566a16-a8a1-4ef0-9f6b-dffa570a1185 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:17.788164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da566a16-a8a1-4ef0-9f6b-dffa570a1185 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:17.789581 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da566a16-a8a1-4ef0-9f6b-dffa570a1185 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:17.807294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 475/945] 10.222.0.16 () {56 vars in 1143 bytes} [Sun Oct 30 00:46:17 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:17.813595 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5602d3e1-7b38-405b-859e-029e637b55d6 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:17.921423 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5602d3e1-7b38-405b-859e-029e637b55d6 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:17.922855 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:17.956392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7b2be2-327c-4f7b-9729-a96d9f005c3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:17.957008 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7b2be2-327c-4f7b-9729-a96d9f005c3c None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:17.957276 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7b2be2-327c-4f7b-9729-a96d9f005c3c None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:17.957554 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7b2be2-327c-4f7b-9729-a96d9f005c3c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:17.967687 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7b2be2-327c-4f7b-9729-a96d9f005c3c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:17.969179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7b2be2-327c-4f7b-9729-a96d9f005c3c None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:17.982902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-1c7b2be2-327c-4f7b-9729-a96d9f005c3c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:46:17.982902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:17.982902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:17.982902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:17.982902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:17.982902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:17.982902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:17.982902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:17.982902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:17.982902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:17.982902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:17.982902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:17.982902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:46:17.982902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:46:17.982902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:46:17.982902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:46:17.984080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:17.984080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:17.984080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:17.984080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:17.984080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:17.984080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:17.984080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:17.984080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:17.984080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:17.984080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:17.984080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:17.984080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:17.984080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:17.984080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:17.984080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:17.984080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:17.984080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:46:17.985176 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:46:17.985176 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:46:17.985176 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:46:17.985176 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 471/946] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:46:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:46:17.990640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da566a16-a8a1-4ef0-9f6b-dffa570a1185 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:18.107283 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da566a16-a8a1-4ef0-9f6b-dffa570a1185 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:18.132605 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:18.149163 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23487282-b54c-498a-9eda-33ea3e8d3bb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:18.149861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23487282-b54c-498a-9eda-33ea3e8d3bb0 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:18.150170 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23487282-b54c-498a-9eda-33ea3e8d3bb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:18.150452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23487282-b54c-498a-9eda-33ea3e8d3bb0 None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:18.151100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23487282-b54c-498a-9eda-33ea3e8d3bb0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:18.151398 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23487282-b54c-498a-9eda-33ea3e8d3bb0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:18.153099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23487282-b54c-498a-9eda-33ea3e8d3bb0 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:18.166606 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 476/947] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:46:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:18.173332 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7b2be2-327c-4f7b-9729-a96d9f005c3c None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:18.286874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7b2be2-327c-4f7b-9729-a96d9f005c3c None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:18.324735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc58b239-0f91-48d2-8011-d42154278b5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:18.325367 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc58b239-0f91-48d2-8011-d42154278b5b None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:18.325643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc58b239-0f91-48d2-8011-d42154278b5b None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:18.325914 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc58b239-0f91-48d2-8011-d42154278b5b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:18.333256 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc58b239-0f91-48d2-8011-d42154278b5b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:18.335358 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc58b239-0f91-48d2-8011-d42154278b5b None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:18.349162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-cc58b239-0f91-48d2-8011-d42154278b5b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:46:18.349162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:18.349162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:18.349162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:18.349162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:18.349162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:18.349162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:18.349162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:18.349162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:18.349162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:18.349162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:18.349162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:18.349162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:46:18.349162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:46:18.349162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:46:18.350333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:46:18.350333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:18.350333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:18.350333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:18.350333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:18.350333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:18.350333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:18.350333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:18.350333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:18.350333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:18.350333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:18.350333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:18.350333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:18.350333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:18.350333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:18.350333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:18.350333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:18.356512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:46:18.356512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:46:18.356512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:46:18.356512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:46:18.356512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:46:18.356512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:46:18.356512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:46:18.356512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:46:18.356512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 472/948] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:46:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:46:18.357122 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23487282-b54c-498a-9eda-33ea3e8d3bb0 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:18.468831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23487282-b54c-498a-9eda-33ea3e8d3bb0 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:18.509923 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410ae57c-bef9-49b2-bcf4-3a97a9b890dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:18.510697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410ae57c-bef9-49b2-bcf4-3a97a9b890dd None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:18.511091 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410ae57c-bef9-49b2-bcf4-3a97a9b890dd None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:18.511411 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410ae57c-bef9-49b2-bcf4-3a97a9b890dd None admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:18.512275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410ae57c-bef9-49b2-bcf4-3a97a9b890dd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:18.512586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410ae57c-bef9-49b2-bcf4-3a97a9b890dd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:18.514034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410ae57c-bef9-49b2-bcf4-3a97a9b890dd None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:18.527387 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 477/949] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:46:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:18.533547 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc58b239-0f91-48d2-8011-d42154278b5b None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:18.553499 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:18.645366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc58b239-0f91-48d2-8011-d42154278b5b None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:18.680487 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b36e80-2613-4481-b00e-48943f499b97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:18.681168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b36e80-2613-4481-b00e-48943f499b97 None admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:18.681440 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b36e80-2613-4481-b00e-48943f499b97 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:18.681707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b36e80-2613-4481-b00e-48943f499b97 None admin] PATH_INFO: `/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/63cc035057064805a95e9f12714ebade/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:18.683549 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b36e80-2613-4481-b00e-48943f499b97 None admin] RBAC: Authorizing `identity:create_grant(project_id=658799ed31d2472fb36af4aff0f03aab, user_id=63cc035057064805a95e9f12714ebade, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:18.685018 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b36e80-2613-4481-b00e-48943f499b97 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:18.701647 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 473/950] 10.222.0.16 () {58 vars in 1439 bytes} [Sun Oct 30 00:46:18 2022] PUT /identity/v3/projects/658799ed31d2472fb36af4aff0f03aab/users/63cc035057064805a95e9f12714ebade/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:46:20.886186 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6b141b-11c5-4750-949c-b33df54bd93d None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:20.886788 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6b141b-11c5-4750-949c-b33df54bd93d None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:20.886788 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6b141b-11c5-4750-949c-b33df54bd93d None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:20.888148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 478/951] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:46:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:46:20.893557 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8199750-bd1a-406c-b6d9-dbadc0c1a884 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:20.893987 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8199750-bd1a-406c-b6d9-dbadc0c1a884 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:20.894084 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8199750-bd1a-406c-b6d9-dbadc0c1a884 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:20.913392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-a8199750-bd1a-406c-b6d9-dbadc0c1a884 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:46:20.914580 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a8199750-bd1a-406c-b6d9-dbadc0c1a884 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:46:20.986951 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 474/952] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:46:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 30 00:46:21.277514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6b141b-11c5-4750-949c-b33df54bd93d None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:21.327491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:21.403772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6b141b-11c5-4750-949c-b33df54bd93d None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:21.444068 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2fda6c-c187-4238-9ebd-2641ac5d7d72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:21.444797 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2fda6c-c187-4238-9ebd-2641ac5d7d72 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:21.445199 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2fda6c-c187-4238-9ebd-2641ac5d7d72 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:21.445500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2fda6c-c187-4238-9ebd-2641ac5d7d72 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:21.457251 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2fda6c-c187-4238-9ebd-2641ac5d7d72 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 30 00:46:21.462782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-cf2fda6c-c187-4238-9ebd-2641ac5d7d72 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:46:21.462782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:21.462782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:21.462782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:21.462782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:21.462782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:21.462782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:21.462782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:21.462782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:21.462782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:21.462782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:21.462782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:21.462782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 30 00:46:21.462782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 30 00:46:21.462782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 30 00:46:21.462782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 30 00:46:21.464044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 30 00:46:21.464044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 30 00:46:21.464044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 30 00:46:21.464044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 30 00:46:21.464044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:21.464044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:21.464044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:21.464044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:21.464044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:21.464044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:21.464044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:21.464044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:21.464044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:21.464044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:21.464044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:21.464044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:21.465049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:21.465049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:21.465049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:21.465049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:21.465049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 30 00:46:21.465049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 30 00:46:21.465049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 30 00:46:21.465049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 30 00:46:21.465049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 30 00:46:21.465049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 30 00:46:21.465049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 30 00:46:21.465049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:46:21.465049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 479/953] 10.222.0.16 () {56 vars in 1096 bytes} [Sun Oct 30 00:46:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:46:21.470813 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8199750-bd1a-406c-b6d9-dbadc0c1a884 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:21.475013 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:21.594950 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8199750-bd1a-406c-b6d9-dbadc0c1a884 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:21.633010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe83f04-2c9c-42a8-9d8e-0a2de4116d2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:21.633984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe83f04-2c9c-42a8-9d8e-0a2de4116d2f None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:21.634296 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe83f04-2c9c-42a8-9d8e-0a2de4116d2f None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:21.634632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe83f04-2c9c-42a8-9d8e-0a2de4116d2f None admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:21.635433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe83f04-2c9c-42a8-9d8e-0a2de4116d2f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:21.635875 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe83f04-2c9c-42a8-9d8e-0a2de4116d2f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:21.637440 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe83f04-2c9c-42a8-9d8e-0a2de4116d2f None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:21.646818 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 475/954] 10.222.0.16 () {56 vars in 1099 bytes} [Sun Oct 30 00:46:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:21.653472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2fda6c-c187-4238-9ebd-2641ac5d7d72 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:21.762585 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:21.765671 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2fda6c-c187-4238-9ebd-2641ac5d7d72 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:21.804912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f48965a-1db3-451a-8439-a9f4f763635f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:21.805570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f48965a-1db3-451a-8439-a9f4f763635f None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:21.805864 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f48965a-1db3-451a-8439-a9f4f763635f None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:21.806132 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f48965a-1db3-451a-8439-a9f4f763635f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:21.817164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f48965a-1db3-451a-8439-a9f4f763635f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:21.819033 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f48965a-1db3-451a-8439-a9f4f763635f None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:21.835070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-7f48965a-1db3-451a-8439-a9f4f763635f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:46:21.835070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:21.835070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:21.835070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:21.835070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:21.835070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:21.835070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:21.835070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:21.835070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:21.835070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:21.835070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:21.835070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:21.835070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:46:21.835070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:46:21.835070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:46:21.835070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:46:21.836288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:21.836288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:21.836288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:21.836288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:21.836288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:21.836288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:21.836288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:21.836288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:21.836288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:21.836288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:21.836288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:21.836288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:21.836288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:21.836288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:21.836288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:21.836288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:21.836288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:46:21.837292 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:46:21.837292 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:46:21.837292 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:46:21.837292 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 480/955] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:46:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:46:21.842558 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe83f04-2c9c-42a8-9d8e-0a2de4116d2f None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:21.914378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:21.955844 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe83f04-2c9c-42a8-9d8e-0a2de4116d2f None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:21.992909 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f96b5c-18fb-4523-9cb4-0ee88ad597aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:21.993487 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f96b5c-18fb-4523-9cb4-0ee88ad597aa None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:21.993772 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f96b5c-18fb-4523-9cb4-0ee88ad597aa None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:21.994039 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f96b5c-18fb-4523-9cb4-0ee88ad597aa None admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:21.994675 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f96b5c-18fb-4523-9cb4-0ee88ad597aa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:21.994991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f96b5c-18fb-4523-9cb4-0ee88ad597aa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:21.996370 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f96b5c-18fb-4523-9cb4-0ee88ad597aa None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:22.009609 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 476/956] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:46:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:22.015409 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f48965a-1db3-451a-8439-a9f4f763635f None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:22.137466 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f48965a-1db3-451a-8439-a9f4f763635f None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:22.177029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967da2c0-5b5d-4c9a-a6c1-0a3d9e3dfcca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:22.177959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967da2c0-5b5d-4c9a-a6c1-0a3d9e3dfcca None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:22.178320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967da2c0-5b5d-4c9a-a6c1-0a3d9e3dfcca None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:22.178803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967da2c0-5b5d-4c9a-a6c1-0a3d9e3dfcca None admin] PATH_INFO: `/v3/users/placement` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:22.189374 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967da2c0-5b5d-4c9a-a6c1-0a3d9e3dfcca None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:22.191244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967da2c0-5b5d-4c9a-a6c1-0a3d9e3dfcca None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:22.205471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-967da2c0-5b5d-4c9a-a6c1-0a3d9e3dfcca None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 30 00:46:22.205471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:22.205471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:22.205471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:22.205471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:22.205471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:22.205471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:22.205471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:22.205471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:22.205471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:22.205471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:22.205471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:22.205471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 30 00:46:22.205471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 30 00:46:22.205471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 30 00:46:22.205471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 30 00:46:22.206633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:22.206633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:22.206633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 30 00:46:22.206633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:46:22.206633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 30 00:46:22.206633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 30 00:46:22.206633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:22.206633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:22.206633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:22.206633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:22.206633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:22.206633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:22.206633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:22.206633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:22.206633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:22.206633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:22.206633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:22.206633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:22.208066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:22.208066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:22.208066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 30 00:46:22.208066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 30 00:46:22.208066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 30 00:46:22.208066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 30 00:46:22.208066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 30 00:46:22.208066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 30 00:46:22.208066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 30 00:46:22.208066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:46:22.208066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 481/957] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:46:22 2022] GET /identity/v3/users/placement => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:46:22.212931 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f96b5c-18fb-4523-9cb4-0ee88ad597aa None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:22.336347 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f96b5c-18fb-4523-9cb4-0ee88ad597aa None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:22.367340 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:22.378061 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7756c98a-fc14-4476-9ad8-679bb2533905 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:22.378703 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7756c98a-fc14-4476-9ad8-679bb2533905 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:22.378953 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7756c98a-fc14-4476-9ad8-679bb2533905 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:22.379218 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7756c98a-fc14-4476-9ad8-679bb2533905 None admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:22.379987 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7756c98a-fc14-4476-9ad8-679bb2533905 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:22.380290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7756c98a-fc14-4476-9ad8-679bb2533905 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:22.381823 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7756c98a-fc14-4476-9ad8-679bb2533905 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:22.400354 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 477/958] 10.222.0.16 () {56 vars in 1143 bytes} [Sun Oct 30 00:46:22 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:46:22.406225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967da2c0-5b5d-4c9a-a6c1-0a3d9e3dfcca None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:22.422848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:22.509346 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967da2c0-5b5d-4c9a-a6c1-0a3d9e3dfcca None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:22.545902 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779f2316-b15e-44bf-b1fb-e2ee9c2f740e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:22.546548 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779f2316-b15e-44bf-b1fb-e2ee9c2f740e None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:22.546876 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779f2316-b15e-44bf-b1fb-e2ee9c2f740e None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:22.547143 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779f2316-b15e-44bf-b1fb-e2ee9c2f740e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:22.618068 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779f2316-b15e-44bf-b1fb-e2ee9c2f740e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:22.619540 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779f2316-b15e-44bf-b1fb-e2ee9c2f740e None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:22.632931 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:22.721967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 482/959] 10.222.0.16 () {58 vars in 1282 bytes} [Sun Oct 30 00:46:22 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 30 00:46:22.728061 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7756c98a-fc14-4476-9ad8-679bb2533905 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:22.838001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7756c98a-fc14-4476-9ad8-679bb2533905 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:22.994465 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a931ce-8e54-4f0d-a011-062fb805b4d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:22.995219 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a931ce-8e54-4f0d-a011-062fb805b4d3 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:22.995490 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a931ce-8e54-4f0d-a011-062fb805b4d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:22.995657 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a931ce-8e54-4f0d-a011-062fb805b4d3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:23.006644 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a931ce-8e54-4f0d-a011-062fb805b4d3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:23.008141 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a931ce-8e54-4f0d-a011-062fb805b4d3 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:23.022843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-25a931ce-8e54-4f0d-a011-062fb805b4d3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:46:23.022843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:23.022843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:23.022843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:23.022843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:23.022843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:23.022843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:23.022843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:23.022843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:23.022843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:23.022843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:23.022843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:23.022843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 30 00:46:23.022843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 30 00:46:23.022843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 30 00:46:23.022843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 30 00:46:23.024598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:23.024598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:23.024598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:23.024598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:23.024598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:23.024598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:23.024598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:23.024598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:23.024598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:23.024598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:23.024598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:23.024598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:23.024598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:23.024598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:23.024598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:23.024598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:23.024598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 30 00:46:23.025515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 30 00:46:23.025515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 30 00:46:23.025515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:46:23.025515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 478/960] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:46:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 30 00:46:23.030234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779f2316-b15e-44bf-b1fb-e2ee9c2f740e None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:23.152650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779f2316-b15e-44bf-b1fb-e2ee9c2f740e None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:23.167866 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:23.190865 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3d7bf6-e8d3-46ac-adcd-a2526b7130bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:23.191473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3d7bf6-e8d3-46ac-adcd-a2526b7130bb None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:23.191776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3d7bf6-e8d3-46ac-adcd-a2526b7130bb None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:23.192049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3d7bf6-e8d3-46ac-adcd-a2526b7130bb None admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:23.192725 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3d7bf6-e8d3-46ac-adcd-a2526b7130bb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:23.193158 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3d7bf6-e8d3-46ac-adcd-a2526b7130bb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:23.194826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3d7bf6-e8d3-46ac-adcd-a2526b7130bb None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:23.208848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 483/961] 10.222.0.16 () {56 vars in 1109 bytes} [Sun Oct 30 00:46:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:23.214985 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a931ce-8e54-4f0d-a011-062fb805b4d3 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:23.258163 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:23.343932 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a931ce-8e54-4f0d-a011-062fb805b4d3 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:23.386563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6600d4-a68b-41ed-ba1a-7a282e212fc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:23.387207 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6600d4-a68b-41ed-ba1a-7a282e212fc2 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:23.387480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6600d4-a68b-41ed-ba1a-7a282e212fc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:23.387805 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6600d4-a68b-41ed-ba1a-7a282e212fc2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:23.395394 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6600d4-a68b-41ed-ba1a-7a282e212fc2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:23.397132 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6600d4-a68b-41ed-ba1a-7a282e212fc2 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:23.411010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application [None req-1d6600d4-a68b-41ed-ba1a-7a282e212fc2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:46:23.411010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:23.411010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:23.411010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:23.411010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:23.411010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:23.411010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:23.411010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:23.411010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:23.411010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:23.411010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:23.411010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:23.411010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 30 00:46:23.411010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 30 00:46:23.411010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 30 00:46:23.412225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 30 00:46:23.412225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:23.412225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:23.412225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:23.412225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:23.412225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:23.412225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:23.412225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:23.412225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:23.412225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:23.412225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:23.412225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:23.412225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:23.412225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:23.412225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:23.412225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:23.412225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:23.413106 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 30 00:46:23.413106 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 30 00:46:23.413106 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 30 00:46:23.413106 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 30 00:46:23.413106 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 30 00:46:23.413106 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 30 00:46:23.413106 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 30 00:46:23.413106 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: ERROR keystone.server.flask.application Oct 30 00:46:23.413106 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 479/962] 10.222.0.16 () {56 vars in 1111 bytes} [Sun Oct 30 00:46:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:46:23.418425 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3d7bf6-e8d3-46ac-adcd-a2526b7130bb None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:23.540559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3d7bf6-e8d3-46ac-adcd-a2526b7130bb None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:23.578762 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dadb104-d86b-4ddc-86ed-ec54449bdd77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:23.579398 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dadb104-d86b-4ddc-86ed-ec54449bdd77 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:23.579673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dadb104-d86b-4ddc-86ed-ec54449bdd77 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:23.579983 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dadb104-d86b-4ddc-86ed-ec54449bdd77 None admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:23.580845 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dadb104-d86b-4ddc-86ed-ec54449bdd77 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:23.581150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dadb104-d86b-4ddc-86ed-ec54449bdd77 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:23.582631 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dadb104-d86b-4ddc-86ed-ec54449bdd77 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:23.596508 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 484/963] 10.222.0.16 () {56 vars in 1148 bytes} [Sun Oct 30 00:46:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:23.603333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6600d4-a68b-41ed-ba1a-7a282e212fc2 None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:23.714459 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:23.721183 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6600d4-a68b-41ed-ba1a-7a282e212fc2 None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:23.757491 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b9839d-a964-49c8-bdb7-d9fe883c1b57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:23.758121 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b9839d-a964-49c8-bdb7-d9fe883c1b57 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:23.758394 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b9839d-a964-49c8-bdb7-d9fe883c1b57 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:23.758656 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b9839d-a964-49c8-bdb7-d9fe883c1b57 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:23.759395 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b9839d-a964-49c8-bdb7-d9fe883c1b57 None admin] RBAC: Adding query filter params (role.id=35f59686b5d84291a8d9baf19287eb5a, scope.project.id=658799ed31d2472fb36af4aff0f03aab, user.id=63cc035057064805a95e9f12714ebade) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:23.759694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b9839d-a964-49c8-bdb7-d9fe883c1b57 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:23.761185 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b9839d-a964-49c8-bdb7-d9fe883c1b57 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:23.767738 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 480/964] 10.222.0.16 () {56 vars in 1374 bytes} [Sun Oct 30 00:46:23 2022] GET /identity/v3/role_assignments?user.id=63cc035057064805a95e9f12714ebade&scope.project.id=658799ed31d2472fb36af4aff0f03aab&role.id=35f59686b5d84291a8d9baf19287eb5a => generated 608 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:25.974538 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b82a67b-f582-4dab-b43b-a66d13dd0a90 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:25.975186 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b82a67b-f582-4dab-b43b-a66d13dd0a90 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:25.975186 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b82a67b-f582-4dab-b43b-a66d13dd0a90 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:25.977297 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 485/965] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:46:25 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:46:25.986294 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c603c2-88b4-4fdc-ad31-7451a8a3e2c0 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:25.986699 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c603c2-88b4-4fdc-ad31-7451a8a3e2c0 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:25.986870 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c603c2-88b4-4fdc-ad31-7451a8a3e2c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:26.013256 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-c2c603c2-88b4-4fdc-ad31-7451a8a3e2c0 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:46:26.014429 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-c2c603c2-88b4-4fdc-ad31-7451a8a3e2c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:46:26.088689 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 481/966] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:46:25 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:46:26.377653 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b82a67b-f582-4dab-b43b-a66d13dd0a90 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:26.412893 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:26.488508 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b82a67b-f582-4dab-b43b-a66d13dd0a90 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:26.525958 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbdae4f-c8d5-47a4-a74c-44544658d963 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:26.526566 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbdae4f-c8d5-47a4-a74c-44544658d963 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:26.526832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbdae4f-c8d5-47a4-a74c-44544658d963 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:26.527097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbdae4f-c8d5-47a4-a74c-44544658d963 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:26.527660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbdae4f-c8d5-47a4-a74c-44544658d963 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:26.529042 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbdae4f-c8d5-47a4-a74c-44544658d963 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:26.537833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-bdbdae4f-c8d5-47a4-a74c-44544658d963 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 30 00:46:26.537833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:26.537833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:46:26.537833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:46:26.537833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:46:26.537833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:46:26.537833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:46:26.537833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:46:26.537833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 30 00:46:26.537833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:46:26.537833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:46:26.537833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:46:26.537833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:26.537833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:26.537833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:26.537833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:26.537833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:26.539035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:26.539035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:26.539035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:26.539035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:26.539035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:26.539035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:26.539035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:46:26.539035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:46:26.539035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:46:26.539035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:46:26.539035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:26.539035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:26.539035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:26.539035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:26.539035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:26.539035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:26.540002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:26.540002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:26.540002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:26.540002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:26.540002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:26.540002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:26.540002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:26.540002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:26.540002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:26.540002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:26.540002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:46:26.540002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:46:26.540002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 30 00:46:26.540002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:46:26.540002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 486/967] 10.222.0.16 () {56 vars in 1117 bytes} [Sun Oct 30 00:46:26 2022] GET /identity/v3/services/placement => generated 90 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:46:26.545273 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c603c2-88b4-4fdc-ad31-7451a8a3e2c0 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:26.645025 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:26.656672 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c603c2-88b4-4fdc-ad31-7451a8a3e2c0 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:26.693879 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fab00c-1a5c-4141-b728-eca7736f22c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:26.694467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fab00c-1a5c-4141-b728-eca7736f22c6 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:26.694735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fab00c-1a5c-4141-b728-eca7736f22c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:26.695000 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fab00c-1a5c-4141-b728-eca7736f22c6 None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:26.695620 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fab00c-1a5c-4141-b728-eca7736f22c6 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:26.695942 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fab00c-1a5c-4141-b728-eca7736f22c6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:26.697275 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fab00c-1a5c-4141-b728-eca7736f22c6 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:26.701960 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 482/968] 10.222.0.16 () {56 vars in 1116 bytes} [Sun Oct 30 00:46:26 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:26.708073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbdae4f-c8d5-47a4-a74c-44544658d963 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:26.808924 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:26.815902 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbdae4f-c8d5-47a4-a74c-44544658d963 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:26.850644 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e31da68-bcd7-4720-b9a3-5059cbbfce80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:26.851233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e31da68-bcd7-4720-b9a3-5059cbbfce80 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:26.851502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e31da68-bcd7-4720-b9a3-5059cbbfce80 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:26.851753 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e31da68-bcd7-4720-b9a3-5059cbbfce80 None admin] PATH_INFO: `/v3/services` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:26.852404 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e31da68-bcd7-4720-b9a3-5059cbbfce80 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:26.852700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e31da68-bcd7-4720-b9a3-5059cbbfce80 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:26.854054 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e31da68-bcd7-4720-b9a3-5059cbbfce80 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:26.859120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 487/969] 10.222.0.16 () {56 vars in 1116 bytes} [Sun Oct 30 00:46:26 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:28.971629 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c51209e-87ad-402f-ae55-61f06696c5d1 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:28.972578 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c51209e-87ad-402f-ae55-61f06696c5d1 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:28.972578 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c51209e-87ad-402f-ae55-61f06696c5d1 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:28.974697 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 483/970] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:46:28 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:46:28.980264 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1572c24f-285c-4eb2-a7aa-766e9bb68509 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:28.980703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1572c24f-285c-4eb2-a7aa-766e9bb68509 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:28.980812 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1572c24f-285c-4eb2-a7aa-766e9bb68509 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:29.000585 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-1572c24f-285c-4eb2-a7aa-766e9bb68509 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:46:29.001740 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-1572c24f-285c-4eb2-a7aa-766e9bb68509 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:46:29.073063 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 488/971] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:46:28 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:46:29.347230 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c51209e-87ad-402f-ae55-61f06696c5d1 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:29.458418 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c51209e-87ad-402f-ae55-61f06696c5d1 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:29.496716 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ba2b96-2f26-4ad2-8c56-1635d0164466 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:29.497296 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ba2b96-2f26-4ad2-8c56-1635d0164466 None admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:29.497564 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ba2b96-2f26-4ad2-8c56-1635d0164466 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:29.497827 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ba2b96-2f26-4ad2-8c56-1635d0164466 None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:29.498496 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ba2b96-2f26-4ad2-8c56-1635d0164466 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:29.499856 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ba2b96-2f26-4ad2-8c56-1635d0164466 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:29.517833 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 484/972] 10.222.0.16 () {60 vars in 1141 bytes} [Sun Oct 30 00:46:29 2022] POST /identity/v3/services => generated 246 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:46:31.653792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e21164-dbfa-42a7-a90b-51ecbfe16e52 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:31.654350 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e21164-dbfa-42a7-a90b-51ecbfe16e52 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:31.654350 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e21164-dbfa-42a7-a90b-51ecbfe16e52 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:31.655588 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 489/973] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:46:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:46:31.661911 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51aae1e-60ce-48d6-8f1a-a5059c651efe None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:31.662637 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51aae1e-60ce-48d6-8f1a-a5059c651efe None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:31.662777 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51aae1e-60ce-48d6-8f1a-a5059c651efe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:31.687547 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-d51aae1e-60ce-48d6-8f1a-a5059c651efe None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:46:31.688835 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d51aae1e-60ce-48d6-8f1a-a5059c651efe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:46:31.699752 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:31.771085 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 485/974] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:46:31 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:46:32.038853 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e21164-dbfa-42a7-a90b-51ecbfe16e52 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:32.141641 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:32.153481 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e21164-dbfa-42a7-a90b-51ecbfe16e52 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:32.192080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aca740d-0ae2-46be-adea-7a08c90733b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:32.192702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aca740d-0ae2-46be-adea-7a08c90733b3 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:32.192978 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aca740d-0ae2-46be-adea-7a08c90733b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:32.193382 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aca740d-0ae2-46be-adea-7a08c90733b3 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:32.194034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aca740d-0ae2-46be-adea-7a08c90733b3 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:32.195402 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aca740d-0ae2-46be-adea-7a08c90733b3 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:32.204238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-3aca740d-0ae2-46be-adea-7a08c90733b3 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 30 00:46:32.204238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:32.204238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:46:32.204238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:46:32.204238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:46:32.204238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:46:32.204238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:46:32.204238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:46:32.204238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 30 00:46:32.204238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:46:32.204238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:46:32.204238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:46:32.204238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:32.204238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:32.204238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:32.204238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:32.204238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:32.205497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:32.205497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:32.205497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:32.205497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:32.205497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:32.205497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:32.205497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:46:32.205497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:46:32.205497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:46:32.205497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:46:32.205497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:32.205497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:32.205497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:32.205497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:32.205497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:32.205497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:32.206500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:32.206500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:32.206500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:32.206500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:32.206500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:32.206500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:32.206500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:32.206500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:32.206500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:32.206500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:32.206500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:46:32.206500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:46:32.206500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 30 00:46:32.206500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:46:32.206500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 490/975] 10.222.0.16 () {56 vars in 1117 bytes} [Sun Oct 30 00:46:32 2022] GET /identity/v3/services/placement => generated 90 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:46:32.212474 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51aae1e-60ce-48d6-8f1a-a5059c651efe None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:32.313297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51aae1e-60ce-48d6-8f1a-a5059c651efe None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:32.348955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be70e366-4bc0-4471-856c-eb7b09885c4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:32.349648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be70e366-4bc0-4471-856c-eb7b09885c4d None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:32.350127 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be70e366-4bc0-4471-856c-eb7b09885c4d None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:32.350600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be70e366-4bc0-4471-856c-eb7b09885c4d None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:32.351303 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be70e366-4bc0-4471-856c-eb7b09885c4d None admin] RBAC: Adding query filter params (name=placement) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:32.351607 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be70e366-4bc0-4471-856c-eb7b09885c4d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:32.353022 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be70e366-4bc0-4471-856c-eb7b09885c4d None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:32.358317 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 486/976] 10.222.0.16 () {56 vars in 1116 bytes} [Sun Oct 30 00:46:32 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:32.364619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aca740d-0ae2-46be-adea-7a08c90733b3 None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:32.467062 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aca740d-0ae2-46be-adea-7a08c90733b3 None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:32.504545 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e29712-96f9-4d00-b808-6efa7a736522 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:32.505136 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e29712-96f9-4d00-b808-6efa7a736522 None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:32.505406 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e29712-96f9-4d00-b808-6efa7a736522 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:32.505669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e29712-96f9-4d00-b808-6efa7a736522 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:32.506352 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e29712-96f9-4d00-b808-6efa7a736522 None admin] RBAC: Adding query filter params (interface=public, service_id=f3b983897ff94752bf11f94e027f299b, region_id=RegionOne) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:32.506649 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e29712-96f9-4d00-b808-6efa7a736522 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:32.508033 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e29712-96f9-4d00-b808-6efa7a736522 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:32.513108 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 491/977] 10.222.0.16 () {56 vars in 1251 bytes} [Sun Oct 30 00:46:32 2022] GET /identity/v3/endpoints?service_id=f3b983897ff94752bf11f94e027f299b&interface=public®ion_id=RegionOne => generated 194 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:46:32.518992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be70e366-4bc0-4471-856c-eb7b09885c4d None admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:32.523118 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:32.636479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be70e366-4bc0-4471-856c-eb7b09885c4d None admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:32.673655 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c36ffae-a20c-406c-bbce-a9e19fa37672 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:32.674251 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c36ffae-a20c-406c-bbce-a9e19fa37672 None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:32.674500 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c36ffae-a20c-406c-bbce-a9e19fa37672 None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:32.674765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c36ffae-a20c-406c-bbce-a9e19fa37672 None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:32.675358 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c36ffae-a20c-406c-bbce-a9e19fa37672 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:32.676749 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c36ffae-a20c-406c-bbce-a9e19fa37672 None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:32.683888 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 487/978] 10.222.0.16 () {56 vars in 1087 bytes} [Sun Oct 30 00:46:32 2022] GET /identity/v3/services => generated 2191 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:46:34.792707 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b495d1-6f4a-445a-b593-6e7e0a3fde3f None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:34.793584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b495d1-6f4a-445a-b593-6e7e0a3fde3f None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:34.793584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b495d1-6f4a-445a-b593-6e7e0a3fde3f None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:34.795610 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 492/979] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:46:34 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:46:34.804096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77ede96-b270-4d83-aba1-7422658568b9 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:34.804496 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77ede96-b270-4d83-aba1-7422658568b9 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:34.804637 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77ede96-b270-4d83-aba1-7422658568b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:34.824267 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-d77ede96-b270-4d83-aba1-7422658568b9 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:46:34.825460 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d77ede96-b270-4d83-aba1-7422658568b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:46:34.889033 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:34.895400 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 488/980] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:46:34 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 30 00:46:35.166501 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b495d1-6f4a-445a-b593-6e7e0a3fde3f None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:35.188658 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:35.275096 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b495d1-6f4a-445a-b593-6e7e0a3fde3f None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:35.309105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb434c4-906b-4f26-b567-758a236ede5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:35.309706 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb434c4-906b-4f26-b567-758a236ede5c None admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:35.309984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb434c4-906b-4f26-b567-758a236ede5c None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:35.310207 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb434c4-906b-4f26-b567-758a236ede5c None admin] PATH_INFO: `/v3/services/placement` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:35.310797 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb434c4-906b-4f26-b567-758a236ede5c None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:35.312173 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb434c4-906b-4f26-b567-758a236ede5c None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:35.321342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application [None req-ccb434c4-906b-4f26-b567-758a236ede5c None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 30 00:46:35.321342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:35.321342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 30 00:46:35.321342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 30 00:46:35.321342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 30 00:46:35.321342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 30 00:46:35.321342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 30 00:46:35.321342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:46:35.321342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 30 00:46:35.321342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:46:35.321342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 30 00:46:35.321342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:46:35.321342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 30 00:46:35.321342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 30 00:46:35.321342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 30 00:46:35.321342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 30 00:46:35.321342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 30 00:46:35.322452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 30 00:46:35.322452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 30 00:46:35.322452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 30 00:46:35.322452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 30 00:46:35.322452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 30 00:46:35.322452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 30 00:46:35.322452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 30 00:46:35.322452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 30 00:46:35.322452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 30 00:46:35.322452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 30 00:46:35.322452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 30 00:46:35.322452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 30 00:46:35.322452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 30 00:46:35.322452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 30 00:46:35.322452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 30 00:46:35.322452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.get_or_create( Oct 30 00:46:35.323645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 30 00:46:35.323645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application with Lock( Oct 30 00:46:35.323645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 30 00:46:35.323645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self._enter() Oct 30 00:46:35.323645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 30 00:46:35.323645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 30 00:46:35.323645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 30 00:46:35.323645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application return self.creator() Oct 30 00:46:35.323645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 30 00:46:35.323645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application created_value = creator( Oct 30 00:46:35.323645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 30 00:46:35.323645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 30 00:46:35.323645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 30 00:46:35.323645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: ERROR keystone.server.flask.application Oct 30 00:46:35.323645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 493/981] 10.222.0.16 () {56 vars in 1117 bytes} [Sun Oct 30 00:46:35 2022] GET /identity/v3/services/placement => generated 90 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 30 00:46:35.328658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77ede96-b270-4d83-aba1-7422658568b9 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:35.435218 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77ede96-b270-4d83-aba1-7422658568b9 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:35.470642 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64833037-8fe7-46dd-a85d-d2454a9dad3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:35.471233 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64833037-8fe7-46dd-a85d-d2454a9dad3f None admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:35.471499 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64833037-8fe7-46dd-a85d-d2454a9dad3f None admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:35.471769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64833037-8fe7-46dd-a85d-d2454a9dad3f None admin] PATH_INFO: `/v3/services` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:35.472397 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64833037-8fe7-46dd-a85d-d2454a9dad3f None admin] RBAC: Adding query filter params (name=placement) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:46:35.472686 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64833037-8fe7-46dd-a85d-d2454a9dad3f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:35.474031 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64833037-8fe7-46dd-a85d-d2454a9dad3f None admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:35.479445 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 489/982] 10.222.0.16 () {56 vars in 1116 bytes} [Sun Oct 30 00:46:35 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:46:35.485648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb434c4-906b-4f26-b567-758a236ede5c None admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:46:35.573867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:35.597150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb434c4-906b-4f26-b567-758a236ede5c None admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:46:35.636485 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75be6533-5868-40f2-a6b0-eac68397a992 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:46:35.637071 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75be6533-5868-40f2-a6b0-eac68397a992 None admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:46:35.637344 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75be6533-5868-40f2-a6b0-eac68397a992 None admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:46:35.637615 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75be6533-5868-40f2-a6b0-eac68397a992 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:46:35.638248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75be6533-5868-40f2-a6b0-eac68397a992 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:46:35.639603 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75be6533-5868-40f2-a6b0-eac68397a992 None admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:46:35.660898 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 494/983] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:46:35 2022] POST /identity/v3/endpoints => generated 331 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:46:45.476304 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=69 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:46:45.656667 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=30,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:47:39.333728 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b8589e-773a-438d-a0ab-a4ccf995c70d None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:39.334217 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b8589e-773a-438d-a0ab-a4ccf995c70d None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:39.334314 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b8589e-773a-438d-a0ab-a4ccf995c70d None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:39.339928 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 490/984] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:47:39 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:47:39.343261 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d99d782-1ef7-4908-a539-4f322b10609d None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:39.343711 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d99d782-1ef7-4908-a539-4f322b10609d None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:39.343842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d99d782-1ef7-4908-a539-4f322b10609d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:39.477675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-3d99d782-1ef7-4908-a539-4f322b10609d None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:47:39.478986 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3d99d782-1ef7-4908-a539-4f322b10609d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:47:39.587828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 495/985] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:47:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 248 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:47:39.880421 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71ecf3b-6f52-4979-a75c-8775e6dccc2a None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:39.880867 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71ecf3b-6f52-4979-a75c-8775e6dccc2a None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:39.881263 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71ecf3b-6f52-4979-a75c-8775e6dccc2a None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:39.882736 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 491/986] 10.222.0.16 () {54 vars in 965 bytes} [Sun Oct 30 00:47:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:47:39.889225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64d0b70-feaf-4050-aa19-f24be69efb1f None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:39.889625 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64d0b70-feaf-4050-aa19-f24be69efb1f None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:39.890001 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64d0b70-feaf-4050-aa19-f24be69efb1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:39.982483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-c64d0b70-feaf-4050-aa19-f24be69efb1f None None] MFA Rules not processed for user `81999329edfb4d4cac567d88d1f52909`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:47:39.983976 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c64d0b70-feaf-4050-aa19-f24be69efb1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:47:40.010523 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:47:40.073325 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 496/987] 10.222.0.16 () {58 vars in 1062 bytes} [Sun Oct 30 00:47:39 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:47:40.082754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71ecf3b-6f52-4979-a75c-8775e6dccc2a None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:47:40.084575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-c71ecf3b-6f52-4979-a75c-8775e6dccc2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:47:40.202952 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71ecf3b-6f52-4979-a75c-8775e6dccc2a None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:47:40.279832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57e6da5-6537-4ab6-a13e-9bad8457492f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:47:40.280430 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57e6da5-6537-4ab6-a13e-9bad8457492f service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:40.280669 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57e6da5-6537-4ab6-a13e-9bad8457492f service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:40.280958 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57e6da5-6537-4ab6-a13e-9bad8457492f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:40.282525 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d57e6da5-6537-4ab6-a13e-9bad8457492f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:47:40.340874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:47:40.405576 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57e6da5-6537-4ab6-a13e-9bad8457492f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:47:40.407210 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57e6da5-6537-4ab6-a13e-9bad8457492f service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:47:40.426958 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 492/988] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:47:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:47:42.840495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96aff866-cc6e-48a9-87a0-a15f0dc81b22 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:42.841008 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96aff866-cc6e-48a9-87a0-a15f0dc81b22 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:42.841008 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96aff866-cc6e-48a9-87a0-a15f0dc81b22 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:42.842416 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 497/989] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:47:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:47:42.848023 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9313dad0-fc93-4316-a107-f0057072df41 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:42.848454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9313dad0-fc93-4316-a107-f0057072df41 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:42.848673 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9313dad0-fc93-4316-a107-f0057072df41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:42.920713 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-9313dad0-fc93-4316-a107-f0057072df41 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:47:42.921907 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-9313dad0-fc93-4316-a107-f0057072df41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:47:43.022486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 493/990] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:47:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:47:43.302782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96aff866-cc6e-48a9-87a0-a15f0dc81b22 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:47:43.305210 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-96aff866-cc6e-48a9-87a0-a15f0dc81b22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:47:43.421060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96aff866-cc6e-48a9-87a0-a15f0dc81b22 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:47:43.498325 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:47:43.500898 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c54a281-38d2-423e-b0e0-fbe98203346b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:47:43.501516 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c54a281-38d2-423e-b0e0-fbe98203346b service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:43.501792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c54a281-38d2-423e-b0e0-fbe98203346b service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:43.502066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c54a281-38d2-423e-b0e0-fbe98203346b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:43.504258 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-2c54a281-38d2-423e-b0e0-fbe98203346b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:47:43.615245 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c54a281-38d2-423e-b0e0-fbe98203346b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:47:43.616589 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c54a281-38d2-423e-b0e0-fbe98203346b service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:47:43.635584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 498/991] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:47:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:47:46.080914 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3075a63f-0661-41bd-9717-1ad47d1545af None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:46.081476 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3075a63f-0661-41bd-9717-1ad47d1545af None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:46.081476 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3075a63f-0661-41bd-9717-1ad47d1545af None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:46.082796 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 494/992] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:47:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:47:46.088464 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c499114c-3cca-4299-8f78-d109b97ac1cd None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:46.088902 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c499114c-3cca-4299-8f78-d109b97ac1cd None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:46.088993 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c499114c-3cca-4299-8f78-d109b97ac1cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:46.154615 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-c499114c-3cca-4299-8f78-d109b97ac1cd None None] MFA Rules not processed for user `1467c783ff2d42a9b97f8ca3d67bea0a`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:47:46.155839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c499114c-3cca-4299-8f78-d109b97ac1cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:47:46.208255 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:47:46.253771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 499/993] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:47:46 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:47:46.539333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3075a63f-0661-41bd-9717-1ad47d1545af None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:47:46.541788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-3075a63f-0661-41bd-9717-1ad47d1545af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:47:46.556985 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:47:46.659243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3075a63f-0661-41bd-9717-1ad47d1545af None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:47:46.755626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb1c2ee-bb3c-4a36-81ee-66834f855f19 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:47:46.756262 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb1c2ee-bb3c-4a36-81ee-66834f855f19 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:46.756537 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb1c2ee-bb3c-4a36-81ee-66834f855f19 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:46.756808 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb1c2ee-bb3c-4a36-81ee-66834f855f19 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:46.758392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-2fb1c2ee-bb3c-4a36-81ee-66834f855f19 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:47:46.827494 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:47:46.876199 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb1c2ee-bb3c-4a36-81ee-66834f855f19 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:47:46.877644 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb1c2ee-bb3c-4a36-81ee-66834f855f19 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:47:46.895860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 495/994] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:47:46 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:47:49.588308 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197d6a4d-a92d-48b5-a448-faded87e7ca8 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:49.588932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197d6a4d-a92d-48b5-a448-faded87e7ca8 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:49.588932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197d6a4d-a92d-48b5-a448-faded87e7ca8 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:49.590180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 500/995] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:47:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:47:49.596038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618004b8-4c62-4a5e-bf68-049350813222 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:49.596471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618004b8-4c62-4a5e-bf68-049350813222 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:49.596626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618004b8-4c62-4a5e-bf68-049350813222 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:49.672482 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-618004b8-4c62-4a5e-bf68-049350813222 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:47:49.673664 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-618004b8-4c62-4a5e-bf68-049350813222 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:47:49.771873 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 496/996] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:47:49 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:47:50.051178 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197d6a4d-a92d-48b5-a448-faded87e7ca8 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:47:50.053714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-197d6a4d-a92d-48b5-a448-faded87e7ca8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:47:50.168737 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197d6a4d-a92d-48b5-a448-faded87e7ca8 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:47:50.246537 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679a855c-f5b7-4692-8b85-3e139ea4b8d5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:47:50.247139 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679a855c-f5b7-4692-8b85-3e139ea4b8d5 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:50.247412 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679a855c-f5b7-4692-8b85-3e139ea4b8d5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:50.247676 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679a855c-f5b7-4692-8b85-3e139ea4b8d5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:50.249288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-679a855c-f5b7-4692-8b85-3e139ea4b8d5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:47:50.255972 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:47:50.372903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679a855c-f5b7-4692-8b85-3e139ea4b8d5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:47:50.374339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679a855c-f5b7-4692-8b85-3e139ea4b8d5 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:47:50.392986 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 501/997] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:47:50 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:47:53.206160 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b250949e-b213-469c-bf1e-670c53a99121 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:53.206721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b250949e-b213-469c-bf1e-670c53a99121 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:53.206721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b250949e-b213-469c-bf1e-670c53a99121 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:53.207981 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 497/998] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:47:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:47:53.213668 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d831241-c0cf-4e19-bd1e-be03b3c78947 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:53.214075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d831241-c0cf-4e19-bd1e-be03b3c78947 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:53.214372 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d831241-c0cf-4e19-bd1e-be03b3c78947 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:53.288038 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-0d831241-c0cf-4e19-bd1e-be03b3c78947 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:47:53.289292 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-0d831241-c0cf-4e19-bd1e-be03b3c78947 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:47:53.358466 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:47:53.401215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 502/999] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:47:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:47:53.679514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b250949e-b213-469c-bf1e-670c53a99121 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:47:53.681106 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b250949e-b213-469c-bf1e-670c53a99121 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:47:53.703220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:47:53.787451 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b250949e-b213-469c-bf1e-670c53a99121 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:47:53.862098 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea25625-4b2b-4f33-9764-040901b594b4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:47:53.862734 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea25625-4b2b-4f33-9764-040901b594b4 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:53.862978 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea25625-4b2b-4f33-9764-040901b594b4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:53.863271 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea25625-4b2b-4f33-9764-040901b594b4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:53.864950 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-aea25625-4b2b-4f33-9764-040901b594b4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:47:53.936764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:47:53.977215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea25625-4b2b-4f33-9764-040901b594b4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:47:53.978722 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea25625-4b2b-4f33-9764-040901b594b4 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:47:53.997136 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 498/1000] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:47:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:47:56.174225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c902c79-aef1-4d70-aa1a-ef4f2c6a2e46 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:56.174772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c902c79-aef1-4d70-aa1a-ef4f2c6a2e46 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:56.174772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c902c79-aef1-4d70-aa1a-ef4f2c6a2e46 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:56.176110 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 503/1001] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:47:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:47:56.181563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fd36fd-751c-449b-92c6-ba3db8519375 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:56.181984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fd36fd-751c-449b-92c6-ba3db8519375 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:56.182069 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8fd36fd-751c-449b-92c6-ba3db8519375 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:56.249911 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-f8fd36fd-751c-449b-92c6-ba3db8519375 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:47:56.251096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-f8fd36fd-751c-449b-92c6-ba3db8519375 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:47:56.359358 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 499/1002] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:47:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:47:56.645392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5b15fd-3d3a-4383-bd48-37909e81fb7e None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:56.645630 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5b15fd-3d3a-4383-bd48-37909e81fb7e None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:56.645898 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5b15fd-3d3a-4383-bd48-37909e81fb7e None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:56.647236 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 504/1003] 10.222.0.16 () {54 vars in 965 bytes} [Sun Oct 30 00:47:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:47:56.652978 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb12efac-daee-4628-84e9-2e263923f0f8 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:56.653265 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb12efac-daee-4628-84e9-2e263923f0f8 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:56.653534 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb12efac-daee-4628-84e9-2e263923f0f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:56.686025 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:47:56.733655 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-eb12efac-daee-4628-84e9-2e263923f0f8 None None] MFA Rules not processed for user `81999329edfb4d4cac567d88d1f52909`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:47:56.734865 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-eb12efac-daee-4628-84e9-2e263923f0f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:47:56.815002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 500/1004] 10.222.0.16 () {58 vars in 1062 bytes} [Sun Oct 30 00:47:56 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:47:56.824850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5b15fd-3d3a-4383-bd48-37909e81fb7e None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:47:56.826375 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3b5b15fd-3d3a-4383-bd48-37909e81fb7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:47:56.935494 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5b15fd-3d3a-4383-bd48-37909e81fb7e None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:47:57.013068 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b02438-264f-4a65-ad37-a5091e44c60d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:47:57.013664 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b02438-264f-4a65-ad37-a5091e44c60d service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:57.013906 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b02438-264f-4a65-ad37-a5091e44c60d service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:57.014187 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b02438-264f-4a65-ad37-a5091e44c60d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:57.015750 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-95b02438-264f-4a65-ad37-a5091e44c60d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:47:57.033413 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:47:57.129314 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b02438-264f-4a65-ad37-a5091e44c60d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:47:57.130897 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b02438-264f-4a65-ad37-a5091e44c60d service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:47:57.149917 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 505/1005] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:47:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:47:59.567611 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd987267-375d-4c1b-a642-13cb8f0eca87 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:59.568230 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd987267-375d-4c1b-a642-13cb8f0eca87 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:59.568230 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd987267-375d-4c1b-a642-13cb8f0eca87 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:59.570103 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 501/1006] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:47:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:47:59.575864 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c04f457-c12a-4630-8280-56d732283416 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:47:59.576394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c04f457-c12a-4630-8280-56d732283416 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:47:59.576394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c04f457-c12a-4630-8280-56d732283416 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:47:59.649197 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-5c04f457-c12a-4630-8280-56d732283416 None None] MFA Rules not processed for user `1467c783ff2d42a9b97f8ca3d67bea0a`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:47:59.650412 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-5c04f457-c12a-4630-8280-56d732283416 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:47:59.718829 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:47:59.750612 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 506/1007] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:47:59 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:48:00.027809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd987267-375d-4c1b-a642-13cb8f0eca87 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:48:00.029356 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-dd987267-375d-4c1b-a642-13cb8f0eca87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:00.141084 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd987267-375d-4c1b-a642-13cb8f0eca87 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:48:00.162605 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:00.220061 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09423b3d-8b9d-4f41-bc81-720c2ca15115 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:48:00.220661 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09423b3d-8b9d-4f41-bc81-720c2ca15115 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:00.220902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09423b3d-8b9d-4f41-bc81-720c2ca15115 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:00.221190 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09423b3d-8b9d-4f41-bc81-720c2ca15115 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:00.222749 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-09423b3d-8b9d-4f41-bc81-720c2ca15115 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:00.333928 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09423b3d-8b9d-4f41-bc81-720c2ca15115 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:48:00.335677 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09423b3d-8b9d-4f41-bc81-720c2ca15115 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:48:00.354081 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 502/1008] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:48:00 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:48:02.844208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7799f613-1ad5-41e3-98e0-655dc2965f3c None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:02.844722 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7799f613-1ad5-41e3-98e0-655dc2965f3c None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:02.844809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7799f613-1ad5-41e3-98e0-655dc2965f3c None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:02.846796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 507/1009] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:48:02 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:48:02.854690 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f34c48-3a51-470e-9160-3ebee187604f None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:02.855159 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f34c48-3a51-470e-9160-3ebee187604f None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:02.855159 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f34c48-3a51-470e-9160-3ebee187604f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:02.920964 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:02.924108 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-77f34c48-3a51-470e-9160-3ebee187604f None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:48:02.926098 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-77f34c48-3a51-470e-9160-3ebee187604f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:03.028458 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 503/1010] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:48:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:48:03.324584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7799f613-1ad5-41e3-98e0-655dc2965f3c None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:48:03.326998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-7799f613-1ad5-41e3-98e0-655dc2965f3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:03.435280 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7799f613-1ad5-41e3-98e0-655dc2965f3c None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:48:03.522203 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ee92df-71fe-4ae2-9752-94673d248646 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:48:03.522803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ee92df-71fe-4ae2-9752-94673d248646 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:03.523072 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ee92df-71fe-4ae2-9752-94673d248646 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:03.523333 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ee92df-71fe-4ae2-9752-94673d248646 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:03.524943 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-f6ee92df-71fe-4ae2-9752-94673d248646 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:03.553352 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:03.639099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ee92df-71fe-4ae2-9752-94673d248646 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:48:03.640483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ee92df-71fe-4ae2-9752-94673d248646 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:48:03.659055 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 508/1011] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:48:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:48:05.833980 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e9ad9b-09c1-4ed1-9fab-cb27d19da3d0 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:05.834825 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e9ad9b-09c1-4ed1-9fab-cb27d19da3d0 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:05.834825 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e9ad9b-09c1-4ed1-9fab-cb27d19da3d0 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:05.836995 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 504/1012] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:48:05 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:48:05.845800 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7c9a48-c6ee-4ffc-b013-9ea26f4fe2f9 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:05.846206 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7c9a48-c6ee-4ffc-b013-9ea26f4fe2f9 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:05.846368 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7c9a48-c6ee-4ffc-b013-9ea26f4fe2f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:05.918744 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-bf7c9a48-c6ee-4ffc-b013-9ea26f4fe2f9 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:48:05.919991 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-bf7c9a48-c6ee-4ffc-b013-9ea26f4fe2f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:06.000344 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:06.023470 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 509/1013] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:48:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:48:06.310338 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e9ad9b-09c1-4ed1-9fab-cb27d19da3d0 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:48:06.311977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-26e9ad9b-09c1-4ed1-9fab-cb27d19da3d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:06.438671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e9ad9b-09c1-4ed1-9fab-cb27d19da3d0 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:48:06.470143 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:06.516430 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0bb539-0dd2-4dfc-a249-3924b1c81146 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:48:06.517015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0bb539-0dd2-4dfc-a249-3924b1c81146 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:06.517280 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0bb539-0dd2-4dfc-a249-3924b1c81146 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:06.517543 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0bb539-0dd2-4dfc-a249-3924b1c81146 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:06.519103 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-6c0bb539-0dd2-4dfc-a249-3924b1c81146 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:06.647243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0bb539-0dd2-4dfc-a249-3924b1c81146 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:48:06.648737 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0bb539-0dd2-4dfc-a249-3924b1c81146 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:48:06.667564 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 505/1014] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:48:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:48:09.156184 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f067e5-bc3e-4f18-891a-06cb7e28f948 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:09.156724 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f067e5-bc3e-4f18-891a-06cb7e28f948 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:09.156724 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f067e5-bc3e-4f18-891a-06cb7e28f948 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:09.157982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 510/1015] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:48:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:48:09.164316 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52aef4f4-8856-413b-b725-48002c124ca4 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:09.164767 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52aef4f4-8856-413b-b725-48002c124ca4 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:09.164875 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52aef4f4-8856-413b-b725-48002c124ca4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:09.237779 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-52aef4f4-8856-413b-b725-48002c124ca4 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:48:09.238961 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-52aef4f4-8856-413b-b725-48002c124ca4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:09.247376 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:09.338097 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 506/1016] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:48:09 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:48:09.633799 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f067e5-bc3e-4f18-891a-06cb7e28f948 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:48:09.635939 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-38f067e5-bc3e-4f18-891a-06cb7e28f948 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:09.744042 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f067e5-bc3e-4f18-891a-06cb7e28f948 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:48:09.836232 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96784aef-e072-40d5-8952-d0729512c1a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:48:09.836834 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96784aef-e072-40d5-8952-d0729512c1a8 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:09.837105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96784aef-e072-40d5-8952-d0729512c1a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:09.837374 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96784aef-e072-40d5-8952-d0729512c1a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:09.839019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-96784aef-e072-40d5-8952-d0729512c1a8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:09.876605 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:09.964215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96784aef-e072-40d5-8952-d0729512c1a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:48:09.965644 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96784aef-e072-40d5-8952-d0729512c1a8 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:48:09.986771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 511/1017] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:48:09 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:48:14.141466 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fd0bd7-484f-4ab8-8668-c899910cceae None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:14.142007 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fd0bd7-484f-4ab8-8668-c899910cceae None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:14.142007 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8fd0bd7-484f-4ab8-8668-c899910cceae None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:14.143338 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 507/1018] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:48:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:48:14.148913 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9471980-69ea-4245-85b0-533a59af0ab6 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:14.149327 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9471980-69ea-4245-85b0-533a59af0ab6 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:14.149327 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9471980-69ea-4245-85b0-533a59af0ab6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:14.218321 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-f9471980-69ea-4245-85b0-533a59af0ab6 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:48:14.219454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-f9471980-69ea-4245-85b0-533a59af0ab6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:14.305256 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:14.317938 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 512/1019] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:48:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:48:14.597405 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8fd0bd7-484f-4ab8-8668-c899910cceae None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:48:14.599028 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-c8fd0bd7-484f-4ab8-8668-c899910cceae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:14.708753 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8fd0bd7-484f-4ab8-8668-c899910cceae None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:48:14.747211 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:14.783301 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bba7010-2e3b-4c9d-bdc8-50a38f8de206 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:48:14.783934 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bba7010-2e3b-4c9d-bdc8-50a38f8de206 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:14.784200 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bba7010-2e3b-4c9d-bdc8-50a38f8de206 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:14.784462 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bba7010-2e3b-4c9d-bdc8-50a38f8de206 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:14.786024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-0bba7010-2e3b-4c9d-bdc8-50a38f8de206 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:14.901530 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bba7010-2e3b-4c9d-bdc8-50a38f8de206 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:48:14.902863 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bba7010-2e3b-4c9d-bdc8-50a38f8de206 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:48:14.921622 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 508/1020] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:48:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:48:17.727195 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759f3edd-80fb-47fb-8ba5-94b8765d652b None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:17.727822 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759f3edd-80fb-47fb-8ba5-94b8765d652b None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:17.727822 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759f3edd-80fb-47fb-8ba5-94b8765d652b None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:17.729127 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 513/1021] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:48:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:48:17.734539 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9eb915-14d4-456a-a11f-010fe68dc253 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:17.735322 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9eb915-14d4-456a-a11f-010fe68dc253 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:17.735588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9eb915-14d4-456a-a11f-010fe68dc253 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:17.804957 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-fc9eb915-14d4-456a-a11f-010fe68dc253 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:48:17.806154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-fc9eb915-14d4-456a-a11f-010fe68dc253 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:17.825425 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:17.913491 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 509/1022] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:48:17 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:48:18.202058 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759f3edd-80fb-47fb-8ba5-94b8765d652b None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:48:18.203603 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-759f3edd-80fb-47fb-8ba5-94b8765d652b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:18.309343 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759f3edd-80fb-47fb-8ba5-94b8765d652b None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:48:18.391254 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dbd5ec-f1f0-4381-805f-3d311222b382 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:48:18.391899 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dbd5ec-f1f0-4381-805f-3d311222b382 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:18.392176 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dbd5ec-f1f0-4381-805f-3d311222b382 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:18.392443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dbd5ec-f1f0-4381-805f-3d311222b382 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:18.394027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-f9dbd5ec-f1f0-4381-805f-3d311222b382 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:18.439767 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:18.509532 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9dbd5ec-f1f0-4381-805f-3d311222b382 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:48:18.511796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9dbd5ec-f1f0-4381-805f-3d311222b382 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:48:18.546934 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 514/1023] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:48:18 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:48:22.713411 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d834940-14bf-4a25-be81-5f3e3636fa2a None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:22.714253 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d834940-14bf-4a25-be81-5f3e3636fa2a None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:22.714253 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d834940-14bf-4a25-be81-5f3e3636fa2a None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:22.716364 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 510/1024] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:48:22 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:48:22.722100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871eaf37-bff4-47b5-99ba-cf65dbd12b0e None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:22.722519 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871eaf37-bff4-47b5-99ba-cf65dbd12b0e None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:22.722787 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871eaf37-bff4-47b5-99ba-cf65dbd12b0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:22.798296 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-871eaf37-bff4-47b5-99ba-cf65dbd12b0e None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:48:22.799549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-871eaf37-bff4-47b5-99ba-cf65dbd12b0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:22.901888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:22.904574 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 515/1025] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:48:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:48:23.183276 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d834940-14bf-4a25-be81-5f3e3636fa2a None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:48:23.185777 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-1d834940-14bf-4a25-be81-5f3e3636fa2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:23.299931 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d834940-14bf-4a25-be81-5f3e3636fa2a None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:48:23.353578 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:23.380899 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3a8ccf-87f3-4e6d-94c1-c0d40bacc379 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:48:23.381511 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3a8ccf-87f3-4e6d-94c1-c0d40bacc379 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:23.381756 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3a8ccf-87f3-4e6d-94c1-c0d40bacc379 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:23.382024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3a8ccf-87f3-4e6d-94c1-c0d40bacc379 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:23.383608 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-3f3a8ccf-87f3-4e6d-94c1-c0d40bacc379 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:23.497212 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3a8ccf-87f3-4e6d-94c1-c0d40bacc379 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:48:23.498643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3a8ccf-87f3-4e6d-94c1-c0d40bacc379 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:48:23.520061 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 511/1026] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:48:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:48:28.453505 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9388ad86-256c-4cda-aa28-eddd8676fe0a None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:28.454018 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9388ad86-256c-4cda-aa28-eddd8676fe0a None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:28.454018 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9388ad86-256c-4cda-aa28-eddd8676fe0a None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:28.455759 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 516/1027] 10.222.0.16 () {54 vars in 921 bytes} [Sun Oct 30 00:48:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:48:28.461277 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a4f802-8f85-4ba3-874e-f9fb8e29e60f None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:28.461764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a4f802-8f85-4ba3-874e-f9fb8e29e60f None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:28.461764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a4f802-8f85-4ba3-874e-f9fb8e29e60f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:28.604924 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-77a4f802-8f85-4ba3-874e-f9fb8e29e60f None None] MFA Rules not processed for user `63cc035057064805a95e9f12714ebade`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:48:28.606046 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-77a4f802-8f85-4ba3-874e-f9fb8e29e60f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:28.642879 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:28.753506 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 512/1028] 10.222.0.16 () {58 vars in 1018 bytes} [Sun Oct 30 00:48:28 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 293 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:48:32.852225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235aedd3-593d-4a80-bf8a-e4de9d3c40b1 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:32.852774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235aedd3-593d-4a80-bf8a-e4de9d3c40b1 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:32.853038 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235aedd3-593d-4a80-bf8a-e4de9d3c40b1 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:32.854560 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 517/1029] 10.222.0.16 () {54 vars in 921 bytes} [Sun Oct 30 00:48:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:48:32.860558 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7eda7d-7209-47c7-ba9e-9377e83fe8e6 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:32.861055 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7eda7d-7209-47c7-ba9e-9377e83fe8e6 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:32.861317 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7eda7d-7209-47c7-ba9e-9377e83fe8e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:32.947810 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-7f7eda7d-7209-47c7-ba9e-9377e83fe8e6 None None] MFA Rules not processed for user `63cc035057064805a95e9f12714ebade`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:48:32.949202 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-7f7eda7d-7209-47c7-ba9e-9377e83fe8e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:33.022469 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:33.061476 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 513/1030] 10.222.0.16 () {58 vars in 1018 bytes} [Sun Oct 30 00:48:32 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:48:34.373630 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8920409-f618-449c-9011-999d11edae43 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:34.374173 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8920409-f618-449c-9011-999d11edae43 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:34.374173 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8920409-f618-449c-9011-999d11edae43 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:34.375512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 518/1031] 10.222.0.16 () {54 vars in 921 bytes} [Sun Oct 30 00:48:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:48:34.380628 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34e4bcd-40c5-4c76-91cc-49540464d434 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:34.381036 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34e4bcd-40c5-4c76-91cc-49540464d434 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:34.381170 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34e4bcd-40c5-4c76-91cc-49540464d434 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:34.459983 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-b34e4bcd-40c5-4c76-91cc-49540464d434 None None] MFA Rules not processed for user `63cc035057064805a95e9f12714ebade`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:48:34.461173 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b34e4bcd-40c5-4c76-91cc-49540464d434 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:34.575539 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 514/1032] 10.222.0.16 () {58 vars in 1018 bytes} [Sun Oct 30 00:48:34 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:48:38.086851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b1da5b-3eed-499f-805e-69eff678e6bf None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:38.087365 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b1da5b-3eed-499f-805e-69eff678e6bf None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:38.087365 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b1da5b-3eed-499f-805e-69eff678e6bf None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:38.088856 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 519/1033] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:48:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:48:38.094331 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531fc093-71cf-4ae5-892b-e59c4932af25 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:38.094783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531fc093-71cf-4ae5-892b-e59c4932af25 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:38.094884 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531fc093-71cf-4ae5-892b-e59c4932af25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:38.131931 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:38.168828 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-531fc093-71cf-4ae5-892b-e59c4932af25 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:48:38.170014 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-531fc093-71cf-4ae5-892b-e59c4932af25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:38.280189 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 515/1034] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:48:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:48:38.612117 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefaaa0a-4b55-4637-90f9-f43793933fba None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:38.612698 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefaaa0a-4b55-4637-90f9-f43793933fba None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:38.613147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefaaa0a-4b55-4637-90f9-f43793933fba None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:38.615414 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 520/1035] 10.222.0.16 () {54 vars in 961 bytes} [Sun Oct 30 00:48:38 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:48:38.621331 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7c742c-1548-4838-bd3f-b5c87396b5a2 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:38.621682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7c742c-1548-4838-bd3f-b5c87396b5a2 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:38.622022 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7c742c-1548-4838-bd3f-b5c87396b5a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:38.700744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-eb7c742c-1548-4838-bd3f-b5c87396b5a2 None None] MFA Rules not processed for user `e7a886715b5b4c1d99f0b37c58cb5417`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:48:38.702039 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-eb7c742c-1548-4838-bd3f-b5c87396b5a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:38.752528 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:38.832233 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 516/1036] 10.222.0.16 () {58 vars in 1058 bytes} [Sun Oct 30 00:48:38 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:48:38.840343 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fefaaa0a-4b55-4637-90f9-f43793933fba None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:48:38.841870 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-fefaaa0a-4b55-4637-90f9-f43793933fba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:39.005097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fefaaa0a-4b55-4637-90f9-f43793933fba None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:48:39.081237 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:39.126454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158ccf74-e7f9-4848-8bed-3fcfe6088513 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:48:39.127185 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158ccf74-e7f9-4848-8bed-3fcfe6088513 service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:39.127638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158ccf74-e7f9-4848-8bed-3fcfe6088513 service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:39.127967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158ccf74-e7f9-4848-8bed-3fcfe6088513 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:39.129578 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-158ccf74-e7f9-4848-8bed-3fcfe6088513 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:39.233870 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158ccf74-e7f9-4848-8bed-3fcfe6088513 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:48:39.235355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158ccf74-e7f9-4848-8bed-3fcfe6088513 service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:48:39.254917 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 521/1037] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:48:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:48:41.664596 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59311fb4-de97-4a12-b9b4-20e749e77852 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:41.664596 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59311fb4-de97-4a12-b9b4-20e749e77852 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:41.664596 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59311fb4-de97-4a12-b9b4-20e749e77852 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:41.676121 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d096d6b4-ae90-49a6-a0a4-552f4be5c86a None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:41.676566 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d096d6b4-ae90-49a6-a0a4-552f4be5c86a None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:41.676566 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d096d6b4-ae90-49a6-a0a4-552f4be5c86a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:41.677580 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 517/1038] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:48:41 2022] GET /identity => generated 270 bytes in 18 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:48:41.743934 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-d096d6b4-ae90-49a6-a0a4-552f4be5c86a None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:48:41.745195 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-d096d6b4-ae90-49a6-a0a4-552f4be5c86a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:41.788271 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:41.851757 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 522/1039] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:48:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:48:42.141512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59311fb4-de97-4a12-b9b4-20e749e77852 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:48:42.143066 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-59311fb4-de97-4a12-b9b4-20e749e77852 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:42.315092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59311fb4-de97-4a12-b9b4-20e749e77852 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:48:42.327251 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:42.471952 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7e140a-7bde-4dc0-83e7-23f242a715c9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:48:42.472606 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7e140a-7bde-4dc0-83e7-23f242a715c9 service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:42.472881 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7e140a-7bde-4dc0-83e7-23f242a715c9 service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:42.473147 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7e140a-7bde-4dc0-83e7-23f242a715c9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:42.474835 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-8b7e140a-7bde-4dc0-83e7-23f242a715c9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:42.572772 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7e140a-7bde-4dc0-83e7-23f242a715c9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:48:42.574481 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7e140a-7bde-4dc0-83e7-23f242a715c9 service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:48:42.593487 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 518/1040] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:48:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 452 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:48:45.098391 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2491a8c-2f3d-4f4a-a248-1d16d6662cb5 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:45.098962 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2491a8c-2f3d-4f4a-a248-1d16d6662cb5 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:45.098962 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2491a8c-2f3d-4f4a-a248-1d16d6662cb5 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:45.100407 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 523/1041] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:48:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:48:45.105750 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c0ad8b-9c65-4cf5-a0f2-cebfbdfaa28b None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:45.106229 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c0ad8b-9c65-4cf5-a0f2-cebfbdfaa28b None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:45.106317 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c0ad8b-9c65-4cf5-a0f2-cebfbdfaa28b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:45.123199 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:45.181688 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-e1c0ad8b-9c65-4cf5-a0f2-cebfbdfaa28b None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:48:45.182986 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-e1c0ad8b-9c65-4cf5-a0f2-cebfbdfaa28b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:45.286025 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 519/1042] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:48:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:48:45.577197 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560fbd4e-c69e-4187-ac47-ef7d0ab254b0 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:45.577522 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560fbd4e-c69e-4187-ac47-ef7d0ab254b0 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:45.577747 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560fbd4e-c69e-4187-ac47-ef7d0ab254b0 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:45.579202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 524/1043] 10.222.0.16 () {54 vars in 961 bytes} [Sun Oct 30 00:48:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:48:45.585014 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7033e5f9-72ff-4f04-8574-ff9204a92e26 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:45.585548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7033e5f9-72ff-4f04-8574-ff9204a92e26 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:45.585870 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7033e5f9-72ff-4f04-8574-ff9204a92e26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:45.668056 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-7033e5f9-72ff-4f04-8574-ff9204a92e26 None None] MFA Rules not processed for user `e7a886715b5b4c1d99f0b37c58cb5417`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:48:45.669407 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-7033e5f9-72ff-4f04-8574-ff9204a92e26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:45.706525 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:45.793024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 520/1044] 10.222.0.16 () {58 vars in 1058 bytes} [Sun Oct 30 00:48:45 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:48:45.800915 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560fbd4e-c69e-4187-ac47-ef7d0ab254b0 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:48:45.802473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-560fbd4e-c69e-4187-ac47-ef7d0ab254b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:45.957116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560fbd4e-c69e-4187-ac47-ef7d0ab254b0 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:48:45.980036 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:46.078448 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e738bf5-30c6-4d7e-8392-c50dd58df432 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:48:46.079233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e738bf5-30c6-4d7e-8392-c50dd58df432 service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:46.079603 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e738bf5-30c6-4d7e-8392-c50dd58df432 service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:46.079989 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e738bf5-30c6-4d7e-8392-c50dd58df432 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:46.081685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-2e738bf5-30c6-4d7e-8392-c50dd58df432 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:46.180812 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e738bf5-30c6-4d7e-8392-c50dd58df432 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:48:46.182803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e738bf5-30c6-4d7e-8392-c50dd58df432 service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:48:46.202640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 525/1045] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:48:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:48:46.265168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c743eb74-dffd-4f08-89a8-ee23bbed369f None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:46.265750 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c743eb74-dffd-4f08-89a8-ee23bbed369f None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:46.265865 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c743eb74-dffd-4f08-89a8-ee23bbed369f None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:46.267240 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 521/1046] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:48:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:48:46.273050 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2eb82d1-2ac2-4d36-b493-4109fb0fb8ee None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:46.273571 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2eb82d1-2ac2-4d36-b493-4109fb0fb8ee None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:46.273858 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2eb82d1-2ac2-4d36-b493-4109fb0fb8ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:46.310384 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:46.354834 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-c2eb82d1-2ac2-4d36-b493-4109fb0fb8ee None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:48:46.356315 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c2eb82d1-2ac2-4d36-b493-4109fb0fb8ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:46.477582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 526/1047] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:48:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:48:46.770735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b12f259-216e-4216-b73c-393912824f7f None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:46.771165 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b12f259-216e-4216-b73c-393912824f7f None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:46.771529 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b12f259-216e-4216-b73c-393912824f7f None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:46.773109 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 522/1048] 10.222.0.16 () {54 vars in 963 bytes} [Sun Oct 30 00:48:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:48:46.778429 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b4d20e-fd45-4375-b6be-646019891e36 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:46.778726 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b4d20e-fd45-4375-b6be-646019891e36 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:46.778996 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b4d20e-fd45-4375-b6be-646019891e36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:46.863599 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-c2b4d20e-fd45-4375-b6be-646019891e36 None None] MFA Rules not processed for user `d57b1c923dff462ab9aaf238f95c6b7f`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:48:46.865072 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c2b4d20e-fd45-4375-b6be-646019891e36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:46.914174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:46.946153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 527/1049] 10.222.0.16 () {58 vars in 1060 bytes} [Sun Oct 30 00:48:46 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:48:46.954247 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b12f259-216e-4216-b73c-393912824f7f None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:48:46.955899 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-4b12f259-216e-4216-b73c-393912824f7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:47.067872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b12f259-216e-4216-b73c-393912824f7f None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:48:47.112276 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:47.149741 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2980a799-29b8-4461-9f1a-1f18ae2dfc37 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:48:47.150348 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2980a799-29b8-4461-9f1a-1f18ae2dfc37 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:47.150616 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2980a799-29b8-4461-9f1a-1f18ae2dfc37 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:47.150877 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2980a799-29b8-4461-9f1a-1f18ae2dfc37 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:47.152560 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-2980a799-29b8-4461-9f1a-1f18ae2dfc37 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:47.289228 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2980a799-29b8-4461-9f1a-1f18ae2dfc37 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:48:47.291903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2980a799-29b8-4461-9f1a-1f18ae2dfc37 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:48:47.313295 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 523/1050] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:48:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:48:48.646710 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae76dbb-c35b-4469-8362-ea2ad162d2b3 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:48.647150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae76dbb-c35b-4469-8362-ea2ad162d2b3 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:48.647262 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae76dbb-c35b-4469-8362-ea2ad162d2b3 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:48.648613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 528/1051] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:48:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:48:48.654780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a04758c-1ad9-4bbc-982c-2f3a3e4ec4e0 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:48.655446 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a04758c-1ad9-4bbc-982c-2f3a3e4ec4e0 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:48.655590 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a04758c-1ad9-4bbc-982c-2f3a3e4ec4e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:48.727048 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-8a04758c-1ad9-4bbc-982c-2f3a3e4ec4e0 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:48:48.728320 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-8a04758c-1ad9-4bbc-982c-2f3a3e4ec4e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:48.750325 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:48.837979 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 524/1052] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:48:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:48:49.128243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae76dbb-c35b-4469-8362-ea2ad162d2b3 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:48:49.129789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-6ae76dbb-c35b-4469-8362-ea2ad162d2b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:49.287569 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae76dbb-c35b-4469-8362-ea2ad162d2b3 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:48:49.341408 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:49.413933 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1484752d-5974-4871-b437-8a08f281126b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:48:49.414531 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1484752d-5974-4871-b437-8a08f281126b service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:49.414804 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1484752d-5974-4871-b437-8a08f281126b service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:49.415068 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1484752d-5974-4871-b437-8a08f281126b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:49.416705 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-1484752d-5974-4871-b437-8a08f281126b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:49.509447 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1484752d-5974-4871-b437-8a08f281126b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:48:49.511231 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1484752d-5974-4871-b437-8a08f281126b service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:48:49.530722 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 529/1053] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:48:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:48:51.804178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c06b91-f66b-47b3-baa5-038111c381a4 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:51.804716 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c06b91-f66b-47b3-baa5-038111c381a4 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:51.804716 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c06b91-f66b-47b3-baa5-038111c381a4 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:51.806290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 525/1054] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:48:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:48:51.812336 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be326923-861a-43a4-9522-a7e64ae772fe None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:51.812828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be326923-861a-43a4-9522-a7e64ae772fe None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:51.812911 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be326923-861a-43a4-9522-a7e64ae772fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:51.883565 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-be326923-861a-43a4-9522-a7e64ae772fe None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:48:51.884810 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-be326923-861a-43a4-9522-a7e64ae772fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:51.892565 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:51.990876 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 530/1055] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:48:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:48:52.288551 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c06b91-f66b-47b3-baa5-038111c381a4 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:48:52.290182 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-10c06b91-f66b-47b3-baa5-038111c381a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:52.455670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c06b91-f66b-47b3-baa5-038111c381a4 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:48:52.459674 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:52.581554 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5cbbfd-150d-4f9d-986c-a48f9c7b85f2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:48:52.582282 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5cbbfd-150d-4f9d-986c-a48f9c7b85f2 service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:52.582623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5cbbfd-150d-4f9d-986c-a48f9c7b85f2 service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:52.583003 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5cbbfd-150d-4f9d-986c-a48f9c7b85f2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:52.584675 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-1f5cbbfd-150d-4f9d-986c-a48f9c7b85f2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:52.681318 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5cbbfd-150d-4f9d-986c-a48f9c7b85f2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:48:52.683075 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5cbbfd-150d-4f9d-986c-a48f9c7b85f2 service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:48:52.713899 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 526/1056] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:48:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:48:54.994336 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8157cb42-188c-40a3-aaa0-fc625daf4ed5 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:54.994839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8157cb42-188c-40a3-aaa0-fc625daf4ed5 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:54.994839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8157cb42-188c-40a3-aaa0-fc625daf4ed5 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:54.996553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 531/1057] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:48:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:48:55.002551 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d5032a-8e61-4f4c-8d68-6c18760ee746 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:55.003010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d5032a-8e61-4f4c-8d68-6c18760ee746 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:55.003104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d5032a-8e61-4f4c-8d68-6c18760ee746 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:55.015141 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:55.071226 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-e2d5032a-8e61-4f4c-8d68-6c18760ee746 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:48:55.073101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-e2d5032a-8e61-4f4c-8d68-6c18760ee746 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:55.174701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 527/1058] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:48:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:48:55.465577 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8157cb42-188c-40a3-aaa0-fc625daf4ed5 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:48:55.467114 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-8157cb42-188c-40a3-aaa0-fc625daf4ed5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:55.630517 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:55.643871 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8157cb42-188c-40a3-aaa0-fc625daf4ed5 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:48:55.773386 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6393572-0ecf-4802-8bce-2a88fb8ee567 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:48:55.773986 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6393572-0ecf-4802-8bce-2a88fb8ee567 service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:55.774251 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6393572-0ecf-4802-8bce-2a88fb8ee567 service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:55.774516 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6393572-0ecf-4802-8bce-2a88fb8ee567 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:55.776121 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-a6393572-0ecf-4802-8bce-2a88fb8ee567 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:55.888267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6393572-0ecf-4802-8bce-2a88fb8ee567 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:48:55.889755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6393572-0ecf-4802-8bce-2a88fb8ee567 service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:48:55.899784 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:55.910713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 532/1059] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:48:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 446 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:48:58.288067 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932099b6-cf40-4174-bd08-455b5a76dfa8 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:58.288626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932099b6-cf40-4174-bd08-455b5a76dfa8 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:58.288769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932099b6-cf40-4174-bd08-455b5a76dfa8 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:58.290514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 528/1060] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:48:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:48:58.297522 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a3e941-e424-4752-b953-970f719bb657 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:58.297966 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a3e941-e424-4752-b953-970f719bb657 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:58.298213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a3e941-e424-4752-b953-970f719bb657 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:58.380008 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-62a3e941-e424-4752-b953-970f719bb657 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:48:58.381565 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-62a3e941-e424-4752-b953-970f719bb657 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:58.486713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 533/1061] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:48:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:48:58.776334 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932099b6-cf40-4174-bd08-455b5a76dfa8 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:48:58.778083 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-932099b6-cf40-4174-bd08-455b5a76dfa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:58.873609 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:58.944052 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932099b6-cf40-4174-bd08-455b5a76dfa8 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:48:59.067278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2690e15-6f21-4b79-abfb-f8188f20c042 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:48:59.068043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2690e15-6f21-4b79-abfb-f8188f20c042 service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:48:59.068388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2690e15-6f21-4b79-abfb-f8188f20c042 service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:48:59.069001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2690e15-6f21-4b79-abfb-f8188f20c042 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:48:59.071013 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a2690e15-6f21-4b79-abfb-f8188f20c042 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:48:59.125044 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:48:59.172492 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2690e15-6f21-4b79-abfb-f8188f20c042 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:48:59.174102 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2690e15-6f21-4b79-abfb-f8188f20c042 service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:48:59.194911 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 529/1062] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:48:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:49:01.354897 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466272e8-da2e-4b52-a58d-f97b7e1b5f4d None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:01.355502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466272e8-da2e-4b52-a58d-f97b7e1b5f4d None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:01.355601 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466272e8-da2e-4b52-a58d-f97b7e1b5f4d None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:01.357256 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 534/1063] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:49:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:49:01.362887 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2decab9d-5275-44bf-a440-b75a9ee7e12a None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:01.363343 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2decab9d-5275-44bf-a440-b75a9ee7e12a None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:01.363432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2decab9d-5275-44bf-a440-b75a9ee7e12a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:01.432018 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-2decab9d-5275-44bf-a440-b75a9ee7e12a None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:01.433442 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-2decab9d-5275-44bf-a440-b75a9ee7e12a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:01.532817 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 530/1064] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:49:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:49:01.819490 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466272e8-da2e-4b52-a58d-f97b7e1b5f4d None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:49:01.821105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-466272e8-da2e-4b52-a58d-f97b7e1b5f4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:01.900408 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:01.969234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466272e8-da2e-4b52-a58d-f97b7e1b5f4d None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:49:02.081928 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7a7e75-f575-4d14-bce6-583cb7f947c8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:49:02.082484 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7a7e75-f575-4d14-bce6-583cb7f947c8 service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:02.082789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7a7e75-f575-4d14-bce6-583cb7f947c8 service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:02.083204 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7a7e75-f575-4d14-bce6-583cb7f947c8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:02.085069 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-7e7a7e75-f575-4d14-bce6-583cb7f947c8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:02.125466 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:02.177104 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7a7e75-f575-4d14-bce6-583cb7f947c8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:49:02.178697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7a7e75-f575-4d14-bce6-583cb7f947c8 service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:49:02.197460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 535/1065] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:49:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:49:02.595901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd5321d-4ce5-424d-9f43-4bce5fb49ff1 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:02.596503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd5321d-4ce5-424d-9f43-4bce5fb49ff1 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:02.596503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd5321d-4ce5-424d-9f43-4bce5fb49ff1 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:02.597856 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 531/1066] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:49:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:49:02.603399 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c055403f-e2fd-47bb-abab-f0854e746148 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:02.603851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c055403f-e2fd-47bb-abab-f0854e746148 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:02.604082 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c055403f-e2fd-47bb-abab-f0854e746148 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:02.672285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-c055403f-e2fd-47bb-abab-f0854e746148 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:02.673479 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c055403f-e2fd-47bb-abab-f0854e746148 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:02.773059 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 536/1067] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:49:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:49:03.070644 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88194ec5-0899-4ad8-8459-87787a86d59c None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:03.070920 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88194ec5-0899-4ad8-8459-87787a86d59c None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:03.071225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88194ec5-0899-4ad8-8459-87787a86d59c None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:03.072914 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 532/1068] 10.222.0.16 () {54 vars in 962 bytes} [Sun Oct 30 00:49:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:49:03.078702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c659c054-36bb-4b94-9fbe-a439e3f837e0 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:03.078987 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c659c054-36bb-4b94-9fbe-a439e3f837e0 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:03.079261 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c659c054-36bb-4b94-9fbe-a439e3f837e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:03.095621 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:03.161774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-c659c054-36bb-4b94-9fbe-a439e3f837e0 None None] MFA Rules not processed for user `807b123652d245269336fcffaf36478e`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:03.162972 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c659c054-36bb-4b94-9fbe-a439e3f837e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:03.239079 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 537/1069] 10.222.0.16 () {58 vars in 1059 bytes} [Sun Oct 30 00:49:03 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:49:03.248820 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88194ec5-0899-4ad8-8459-87787a86d59c None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:49:03.250415 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-88194ec5-0899-4ad8-8459-87787a86d59c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:03.282241 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:03.360739 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88194ec5-0899-4ad8-8459-87787a86d59c None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:49:03.435430 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af9e7bc-d22c-4da6-81c7-e72a69567a96 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '807b123652d245269336fcffaf36478e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:49:03.436063 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af9e7bc-d22c-4da6-81c7-e72a69567a96 service glance] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:03.436332 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af9e7bc-d22c-4da6-81c7-e72a69567a96 service glance] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:03.436593 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af9e7bc-d22c-4da6-81c7-e72a69567a96 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:03.438161 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-7af9e7bc-d22c-4da6-81c7-e72a69567a96 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:03.515653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:03.549501 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af9e7bc-d22c-4da6-81c7-e72a69567a96 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:49:03.550937 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af9e7bc-d22c-4da6-81c7-e72a69567a96 service glance] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:49:03.570057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 533/1070] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:49:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:49:04.318590 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f96de53-40f6-4e02-8f80-a0ecddeb4704 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:04.319063 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f96de53-40f6-4e02-8f80-a0ecddeb4704 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:04.319170 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f96de53-40f6-4e02-8f80-a0ecddeb4704 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:04.387925 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-9f96de53-40f6-4e02-8f80-a0ecddeb4704 None None] MFA Rules not processed for user `4716e7afe2784e2a9b872996d895c374`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:04.388436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b9a516-1af2-4ced-ae0a-d1d1d6046c67 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:04.388916 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b9a516-1af2-4ced-ae0a-d1d1d6046c67 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:04.389033 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b9a516-1af2-4ced-ae0a-d1d1d6046c67 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:04.389116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-9f96de53-40f6-4e02-8f80-a0ecddeb4704 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:04.390319 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 534/1071] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:49:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:49:04.395517 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f20c3a-51ad-413c-a83a-31d18a54070e None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:04.395820 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f20c3a-51ad-413c-a83a-31d18a54070e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:04.396116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f20c3a-51ad-413c-a83a-31d18a54070e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:04.444685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:04.464604 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-49f20c3a-51ad-413c-a83a-31d18a54070e None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:04.465905 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-49f20c3a-51ad-413c-a83a-31d18a54070e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:04.478035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 538/1072] 10.222.0.16 () {58 vars in 1009 bytes} [Sun Oct 30 00:49:04 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:49:04.525502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ba041f-a73f-4268-868a-5ad2351cb9c8 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:04.525798 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ba041f-a73f-4268-868a-5ad2351cb9c8 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:04.526066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ba041f-a73f-4268-868a-5ad2351cb9c8 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:04.527629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 539/1073] 10.222.0.16 () {54 vars in 943 bytes} [Sun Oct 30 00:49:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:49:04.533466 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf3f304-71e5-4050-b9f6-93f8e7896bb9 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:04.533915 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf3f304-71e5-4050-b9f6-93f8e7896bb9 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:04.534186 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf3f304-71e5-4050-b9f6-93f8e7896bb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:04.575762 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 535/1074] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:49:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:49:04.611754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-3bf3f304-71e5-4050-b9f6-93f8e7896bb9 None None] MFA Rules not processed for user `67f6eff8c0254cc6a210101dd7af988d`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:04.612974 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3bf3f304-71e5-4050-b9f6-93f8e7896bb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:04.719895 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 540/1075] 10.222.0.16 () {58 vars in 1040 bytes} [Sun Oct 30 00:49:04 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:49:04.729030 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f20c3a-51ad-413c-a83a-31d18a54070e None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:49:04.730358 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-49f20c3a-51ad-413c-a83a-31d18a54070e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:04.769945 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:04.833705 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f20c3a-51ad-413c-a83a-31d18a54070e None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:49:04.860686 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf3f304-71e5-4050-b9f6-93f8e7896bb9 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:49:04.862515 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3bf3f304-71e5-4050-b9f6-93f8e7896bb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:04.895501 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:04.947755 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1bdc6a-b9d4-45cc-bf06-361b964b871b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '67f6eff8c0254cc6a210101dd7af988d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:49:04.948373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1bdc6a-b9d4-45cc-bf06-361b964b871b service swift] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:04.948647 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1bdc6a-b9d4-45cc-bf06-361b964b871b service swift] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:04.948957 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1bdc6a-b9d4-45cc-bf06-361b964b871b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:04.950620 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-7f1bdc6a-b9d4-45cc-bf06-361b964b871b service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:04.964597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf3f304-71e5-4050-b9f6-93f8e7896bb9 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:49:05.004080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:05.037358 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1bdc6a-b9d4-45cc-bf06-361b964b871b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:49:05.038903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1bdc6a-b9d4-45cc-bf06-361b964b871b service swift] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:49:05.047281 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 536/1076] 10.222.0.16 () {60 vars in 1384 bytes} [Sun Oct 30 00:49:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 30 00:49:05.084199 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6884ce01-d807-43f3-ac95-4b6b9b29fcd4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:49:05.084816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6884ce01-d807-43f3-ac95-4b6b9b29fcd4 service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:05.085089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6884ce01-d807-43f3-ac95-4b6b9b29fcd4 service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:05.085475 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6884ce01-d807-43f3-ac95-4b6b9b29fcd4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:05.086907 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-6884ce01-d807-43f3-ac95-4b6b9b29fcd4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:05.132128 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:05.186167 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6884ce01-d807-43f3-ac95-4b6b9b29fcd4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:49:05.187961 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6884ce01-d807-43f3-ac95-4b6b9b29fcd4 service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:49:05.208130 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 541/1077] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:49:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:49:07.475282 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4752b9-b996-40fc-b9c1-a964e162901e None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:07.475845 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4752b9-b996-40fc-b9c1-a964e162901e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:07.475845 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4752b9-b996-40fc-b9c1-a964e162901e None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:07.477194 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 537/1078] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:49:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:49:07.482962 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de642791-b6ae-4ec4-bf83-c670c5c77a76 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:07.483410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de642791-b6ae-4ec4-bf83-c670c5c77a76 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:07.483507 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de642791-b6ae-4ec4-bf83-c670c5c77a76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:07.555991 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-de642791-b6ae-4ec4-bf83-c670c5c77a76 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:07.557224 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-de642791-b6ae-4ec4-bf83-c670c5c77a76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:07.676099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 542/1079] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:49:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:49:07.968792 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4752b9-b996-40fc-b9c1-a964e162901e None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:49:07.970315 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-7a4752b9-b996-40fc-b9c1-a964e162901e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:08.141571 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4752b9-b996-40fc-b9c1-a964e162901e None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:49:08.194501 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:08.272260 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2886e34-02a0-4715-b14c-e8a95edf70f8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:49:08.272869 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2886e34-02a0-4715-b14c-e8a95edf70f8 service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:08.273144 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2886e34-02a0-4715-b14c-e8a95edf70f8 service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:08.273412 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2886e34-02a0-4715-b14c-e8a95edf70f8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:08.275007 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-e2886e34-02a0-4715-b14c-e8a95edf70f8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:08.384981 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2886e34-02a0-4715-b14c-e8a95edf70f8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:49:08.387193 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2886e34-02a0-4715-b14c-e8a95edf70f8 service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:49:08.406757 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 538/1080] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:49:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 438 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:49:10.676045 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41779ec6-c3c0-4e74-9456-ee813c1ce0d4 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:10.676624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41779ec6-c3c0-4e74-9456-ee813c1ce0d4 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:10.676624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41779ec6-c3c0-4e74-9456-ee813c1ce0d4 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:10.677852 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 543/1081] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:49:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:49:10.683151 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802dd2da-cea5-49c4-a1f5-9e77472694ce None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:10.684073 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802dd2da-cea5-49c4-a1f5-9e77472694ce None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:10.684689 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802dd2da-cea5-49c4-a1f5-9e77472694ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:10.758487 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-802dd2da-cea5-49c4-a1f5-9e77472694ce None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:10.759707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-802dd2da-cea5-49c4-a1f5-9e77472694ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:10.762790 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:10.864049 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 539/1082] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:49:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:49:11.150859 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41779ec6-c3c0-4e74-9456-ee813c1ce0d4 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:49:11.153246 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-41779ec6-c3c0-4e74-9456-ee813c1ce0d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:11.176230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:11.318794 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41779ec6-c3c0-4e74-9456-ee813c1ce0d4 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:49:11.427623 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:11.453323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5aa6328-8bfc-4afb-876e-d9b9b678dde6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:49:11.453934 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5aa6328-8bfc-4afb-876e-d9b9b678dde6 service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:11.454213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5aa6328-8bfc-4afb-876e-d9b9b678dde6 service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:11.454480 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5aa6328-8bfc-4afb-876e-d9b9b678dde6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:11.456238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-d5aa6328-8bfc-4afb-876e-d9b9b678dde6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:11.554400 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5aa6328-8bfc-4afb-876e-d9b9b678dde6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:49:11.556027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5aa6328-8bfc-4afb-876e-d9b9b678dde6 service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:49:11.575969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 544/1083] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:49:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:49:13.821199 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c069b7cc-02f9-417f-8cc0-d6b2b8cf3d8b None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:13.821776 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c069b7cc-02f9-417f-8cc0-d6b2b8cf3d8b None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:13.821776 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c069b7cc-02f9-417f-8cc0-d6b2b8cf3d8b None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:13.823576 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 540/1084] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:49:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:49:13.829500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9723381c-40d8-47cb-9a92-2ce6e9b9edbb None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:13.829992 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9723381c-40d8-47cb-9a92-2ce6e9b9edbb None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:13.830186 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9723381c-40d8-47cb-9a92-2ce6e9b9edbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:13.935144 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-9723381c-40d8-47cb-9a92-2ce6e9b9edbb None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:13.936427 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-9723381c-40d8-47cb-9a92-2ce6e9b9edbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:13.993155 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:14.044342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 545/1085] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:49:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:49:14.342386 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c069b7cc-02f9-417f-8cc0-d6b2b8cf3d8b None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:49:14.344227 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-c069b7cc-02f9-417f-8cc0-d6b2b8cf3d8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:14.502472 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:14.516533 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c069b7cc-02f9-417f-8cc0-d6b2b8cf3d8b None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:49:14.642765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6a3d8d-0e64-4bbf-badb-b9a503788e78 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:49:14.643523 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6a3d8d-0e64-4bbf-badb-b9a503788e78 service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:14.643834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6a3d8d-0e64-4bbf-badb-b9a503788e78 service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:14.644097 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6a3d8d-0e64-4bbf-badb-b9a503788e78 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:14.645871 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-0c6a3d8d-0e64-4bbf-badb-b9a503788e78 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:14.745471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6a3d8d-0e64-4bbf-badb-b9a503788e78 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:49:14.747208 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6a3d8d-0e64-4bbf-badb-b9a503788e78 service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:49:14.756680 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:14.768006 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 541/1086] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:49:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:49:24.042361 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=19 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:24.765405 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:27.954964 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6afd27-f333-4db8-89aa-54ce708eb799 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:27.955561 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6afd27-f333-4db8-89aa-54ce708eb799 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:27.955832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6afd27-f333-4db8-89aa-54ce708eb799 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:27.958267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 546/1087] 10.222.0.16 () {54 vars in 918 bytes} [Sun Oct 30 00:49:27 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:49:27.964887 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470714e6-3dde-446f-b1c2-299b76f9abeb None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:27.965476 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470714e6-3dde-446f-b1c2-299b76f9abeb None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:27.965848 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470714e6-3dde-446f-b1c2-299b76f9abeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:28.061701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-470714e6-3dde-446f-b1c2-299b76f9abeb None None] MFA Rules not processed for user `63cc035057064805a95e9f12714ebade`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:28.062896 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-470714e6-3dde-446f-b1c2-299b76f9abeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:28.185114 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 542/1088] 10.222.0.16 () {58 vars in 1015 bytes} [Sun Oct 30 00:49:27 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:49:28.203329 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80641013-84d7-4fca-86b5-90a9ff5584e0 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:28.203606 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80641013-84d7-4fca-86b5-90a9ff5584e0 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:28.203904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80641013-84d7-4fca-86b5-90a9ff5584e0 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:28.205331 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 547/1089] 10.222.0.16 () {54 vars in 918 bytes} [Sun Oct 30 00:49:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:49:28.212743 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d82868-0abe-4be1-9103-9f02a035d0bd None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:28.213041 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d82868-0abe-4be1-9103-9f02a035d0bd None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:28.213326 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d82868-0abe-4be1-9103-9f02a035d0bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:28.301514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:28.305312 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-a0d82868-0abe-4be1-9103-9f02a035d0bd None None] MFA Rules not processed for user `e7a886715b5b4c1d99f0b37c58cb5417`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:28.307503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a0d82868-0abe-4be1-9103-9f02a035d0bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:28.443853 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 543/1090] 10.222.0.16 () {58 vars in 1015 bytes} [Sun Oct 30 00:49:28 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:49:31.018711 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a99c3b-48fd-49cf-ba5f-29f3f0a3b067 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:31.019259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a99c3b-48fd-49cf-ba5f-29f3f0a3b067 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:31.019386 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a99c3b-48fd-49cf-ba5f-29f3f0a3b067 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:31.021753 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 548/1091] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 30 00:49:31 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:49:31.030544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f154a6-0942-4f56-9bde-dd931ea8037f None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:31.030942 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f154a6-0942-4f56-9bde-dd931ea8037f None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:31.031101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f154a6-0942-4f56-9bde-dd931ea8037f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:31.111376 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-49f154a6-0942-4f56-9bde-dd931ea8037f None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:31.113259 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-49f154a6-0942-4f56-9bde-dd931ea8037f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:31.176359 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:31.221876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 544/1092] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 30 00:49:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:49:31.516471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a99c3b-48fd-49cf-ba5f-29f3f0a3b067 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:49:31.518016 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-01a99c3b-48fd-49cf-ba5f-29f3f0a3b067 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:31.669649 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a99c3b-48fd-49cf-ba5f-29f3f0a3b067 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:49:31.753283 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:31.796688 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd67d8b-32a7-49b9-ada0-5c7f1ce7a10c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:49:31.797287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd67d8b-32a7-49b9-ada0-5c7f1ce7a10c service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:31.797561 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd67d8b-32a7-49b9-ada0-5c7f1ce7a10c service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:31.797827 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd67d8b-32a7-49b9-ada0-5c7f1ce7a10c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:31.799415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-1fd67d8b-32a7-49b9-ada0-5c7f1ce7a10c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:31.904018 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd67d8b-32a7-49b9-ada0-5c7f1ce7a10c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:49:31.905608 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd67d8b-32a7-49b9-ada0-5c7f1ce7a10c service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:49:31.927100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 549/1093] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:49:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:49:33.049248 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8ccf4d-e97e-46d7-b33f-c164af22991a None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:33.050177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8ccf4d-e97e-46d7-b33f-c164af22991a None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:33.050177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8ccf4d-e97e-46d7-b33f-c164af22991a None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:33.052223 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 545/1094] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 30 00:49:33 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:49:33.060214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2c7a30-e2a1-4a66-9311-5ff7140b8523 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:33.060629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2c7a30-e2a1-4a66-9311-5ff7140b8523 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:33.060733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2c7a30-e2a1-4a66-9311-5ff7140b8523 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:33.144679 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-1f2c7a30-e2a1-4a66-9311-5ff7140b8523 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:33.145800 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-1f2c7a30-e2a1-4a66-9311-5ff7140b8523 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:33.174472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:33.259827 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 550/1095] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 30 00:49:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:49:33.413430 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8ccf4d-e97e-46d7-b33f-c164af22991a None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:49:33.415698 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-ab8ccf4d-e97e-46d7-b33f-c164af22991a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:33.570511 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8ccf4d-e97e-46d7-b33f-c164af22991a None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:49:33.613922 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:33.689733 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d42425-5b76-470d-8e60-3b9aa095cf00 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:49:33.690238 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d42425-5b76-470d-8e60-3b9aa095cf00 service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:33.690507 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d42425-5b76-470d-8e60-3b9aa095cf00 service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:33.690765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d42425-5b76-470d-8e60-3b9aa095cf00 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:33.692357 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-12d42425-5b76-470d-8e60-3b9aa095cf00 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:33.782986 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d42425-5b76-470d-8e60-3b9aa095cf00 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:49:33.784630 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d42425-5b76-470d-8e60-3b9aa095cf00 service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:49:33.808346 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 546/1096] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:49:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:49:34.919265 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c519deba-34bb-45c5-b709-4440e2e66972 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:34.919820 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c519deba-34bb-45c5-b709-4440e2e66972 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:34.919820 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c519deba-34bb-45c5-b709-4440e2e66972 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:34.921177 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 551/1097] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 30 00:49:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:49:34.926359 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842926e8-465e-40e7-934e-63ef46a40f64 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:34.926853 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842926e8-465e-40e7-934e-63ef46a40f64 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:34.926967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842926e8-465e-40e7-934e-63ef46a40f64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:34.987489 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:35.011039 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-842926e8-465e-40e7-934e-63ef46a40f64 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:35.012361 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-842926e8-465e-40e7-934e-63ef46a40f64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:35.117234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 547/1098] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 30 00:49:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:49:35.273337 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c519deba-34bb-45c5-b709-4440e2e66972 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:49:35.275712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c519deba-34bb-45c5-b709-4440e2e66972 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:35.434582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c519deba-34bb-45c5-b709-4440e2e66972 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:49:35.447948 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:35.550257 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5f2a98-fe07-4782-8cc5-015d93485e99 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:49:35.550884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5f2a98-fe07-4782-8cc5-015d93485e99 service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:35.551151 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5f2a98-fe07-4782-8cc5-015d93485e99 service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:35.551415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5f2a98-fe07-4782-8cc5-015d93485e99 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:35.553004 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-8d5f2a98-fe07-4782-8cc5-015d93485e99 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:35.641107 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5f2a98-fe07-4782-8cc5-015d93485e99 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:49:35.642683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5f2a98-fe07-4782-8cc5-015d93485e99 service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:49:35.660706 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 552/1099] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:49:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:49:36.783939 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d53308d-f328-419f-8d77-a774040be680 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:36.784551 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d53308d-f328-419f-8d77-a774040be680 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:36.784551 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d53308d-f328-419f-8d77-a774040be680 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:36.785811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 548/1100] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 30 00:49:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:49:36.791689 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57e81bf-fa96-433a-a3c9-15ba6b8c2901 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:36.792143 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57e81bf-fa96-433a-a3c9-15ba6b8c2901 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:36.792270 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57e81bf-fa96-433a-a3c9-15ba6b8c2901 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:36.819430 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:36.878282 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-b57e81bf-fa96-433a-a3c9-15ba6b8c2901 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:36.879471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-b57e81bf-fa96-433a-a3c9-15ba6b8c2901 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:36.983430 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 553/1101] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 30 00:49:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:49:37.138722 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d53308d-f328-419f-8d77-a774040be680 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:49:37.140202 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-9d53308d-f328-419f-8d77-a774040be680 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:37.262207 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:37.298662 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d53308d-f328-419f-8d77-a774040be680 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:49:37.420281 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b5c747-b492-4aca-b3d6-dd4b3c2d9e27 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:49:37.420784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b5c747-b492-4aca-b3d6-dd4b3c2d9e27 service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:37.421053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b5c747-b492-4aca-b3d6-dd4b3c2d9e27 service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:37.421316 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b5c747-b492-4aca-b3d6-dd4b3c2d9e27 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:37.422888 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b8b5c747-b492-4aca-b3d6-dd4b3c2d9e27 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:37.505650 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:37.520181 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b5c747-b492-4aca-b3d6-dd4b3c2d9e27 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:49:37.521781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b5c747-b492-4aca-b3d6-dd4b3c2d9e27 service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:49:37.540384 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 549/1102] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:49:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:49:38.636421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c646a886-947b-4ecc-b3d9-935faa833746 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:38.637287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c646a886-947b-4ecc-b3d9-935faa833746 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:38.637287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c646a886-947b-4ecc-b3d9-935faa833746 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:38.638597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 554/1103] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 30 00:49:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:49:38.643635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95d1309-d592-4c04-a4e3-5471e29b83e6 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:38.644115 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95d1309-d592-4c04-a4e3-5471e29b83e6 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:38.644220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95d1309-d592-4c04-a4e3-5471e29b83e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:38.735735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-e95d1309-d592-4c04-a4e3-5471e29b83e6 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:38.736970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-e95d1309-d592-4c04-a4e3-5471e29b83e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:38.841741 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 550/1104] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 30 00:49:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:49:38.992936 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c646a886-947b-4ecc-b3d9-935faa833746 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:49:38.994325 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c646a886-947b-4ecc-b3d9-935faa833746 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:39.083116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:39.147365 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c646a886-947b-4ecc-b3d9-935faa833746 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:49:39.280041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b8b94d-c80b-4838-9f69-0a0a0f894a6f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:49:39.280636 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b8b94d-c80b-4838-9f69-0a0a0f894a6f service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:39.280904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b8b94d-c80b-4838-9f69-0a0a0f894a6f service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:39.281160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b8b94d-c80b-4838-9f69-0a0a0f894a6f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:39.282729 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-42b8b94d-c80b-4838-9f69-0a0a0f894a6f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:39.332021 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:39.376951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b8b94d-c80b-4838-9f69-0a0a0f894a6f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:49:39.378533 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b8b94d-c80b-4838-9f69-0a0a0f894a6f service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:49:39.397071 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 555/1105] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:49:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:49:40.438669 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc8a50a-b6d4-4308-b5a0-0f7cb71691f1 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:40.439586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc8a50a-b6d4-4308-b5a0-0f7cb71691f1 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:40.439586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc8a50a-b6d4-4308-b5a0-0f7cb71691f1 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:40.441782 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 551/1106] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 30 00:49:40 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:49:40.450288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6485b1-0a17-4c83-a148-a6c9633cf97b None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:40.450845 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6485b1-0a17-4c83-a148-a6c9633cf97b None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:40.450845 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6485b1-0a17-4c83-a148-a6c9633cf97b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:40.530438 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-5a6485b1-0a17-4c83-a148-a6c9633cf97b None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:40.531625 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-5a6485b1-0a17-4c83-a148-a6c9633cf97b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:40.639637 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 556/1107] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 30 00:49:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:49:40.788985 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc8a50a-b6d4-4308-b5a0-0f7cb71691f1 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:49:40.791266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-1dc8a50a-b6d4-4308-b5a0-0f7cb71691f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:40.850704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:40.953034 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc8a50a-b6d4-4308-b5a0-0f7cb71691f1 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:49:41.076332 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838a1632-9e91-4c10-acaf-9cb11a70d2ec service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:49:41.077153 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838a1632-9e91-4c10-acaf-9cb11a70d2ec service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:41.077353 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838a1632-9e91-4c10-acaf-9cb11a70d2ec service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:41.077648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838a1632-9e91-4c10-acaf-9cb11a70d2ec service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:41.080234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-838a1632-9e91-4c10-acaf-9cb11a70d2ec service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:41.087412 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:41.178728 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838a1632-9e91-4c10-acaf-9cb11a70d2ec service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:49:41.180398 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838a1632-9e91-4c10-acaf-9cb11a70d2ec service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:49:41.198741 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 552/1108] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:49:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:49:42.229803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca51ece-d26c-428f-acb7-0da9fe7e7cae None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:42.230725 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca51ece-d26c-428f-acb7-0da9fe7e7cae None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:42.230725 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca51ece-d26c-428f-acb7-0da9fe7e7cae None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:42.232884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 557/1109] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 30 00:49:42 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:49:42.241458 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6d4039-1bce-466f-b054-eecab99f104b None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:42.241918 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6d4039-1bce-466f-b054-eecab99f104b None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:42.242145 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6d4039-1bce-466f-b054-eecab99f104b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:42.323020 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-dc6d4039-1bce-466f-b054-eecab99f104b None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:42.324564 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-dc6d4039-1bce-466f-b054-eecab99f104b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:42.394543 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:42.434482 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 553/1110] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 30 00:49:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:49:42.586512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca51ece-d26c-428f-acb7-0da9fe7e7cae None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:49:42.588087 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-2ca51ece-d26c-428f-acb7-0da9fe7e7cae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:42.607460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:42.759162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca51ece-d26c-428f-acb7-0da9fe7e7cae None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:49:42.852667 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:42.879976 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0eeba56-f61f-4257-977b-90f655835503 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:49:42.880584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0eeba56-f61f-4257-977b-90f655835503 service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:42.880852 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0eeba56-f61f-4257-977b-90f655835503 service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:42.881114 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0eeba56-f61f-4257-977b-90f655835503 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:42.882670 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-d0eeba56-f61f-4257-977b-90f655835503 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:42.982080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0eeba56-f61f-4257-977b-90f655835503 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:49:42.983706 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0eeba56-f61f-4257-977b-90f655835503 service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:49:43.002534 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 558/1111] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:49:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:49:44.049004 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d22fff-9e49-42c0-9725-424a1fb5773a None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:44.049930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d22fff-9e49-42c0-9725-424a1fb5773a None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:44.049930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d22fff-9e49-42c0-9725-424a1fb5773a None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:44.052076 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 554/1112] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 30 00:49:44 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:49:44.060301 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7c69c2-b5bf-4a32-b558-794525eacab0 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:44.060739 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7c69c2-b5bf-4a32-b558-794525eacab0 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:44.060846 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7c69c2-b5bf-4a32-b558-794525eacab0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:44.149838 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-5d7c69c2-b5bf-4a32-b558-794525eacab0 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:44.151577 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-5d7c69c2-b5bf-4a32-b558-794525eacab0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:44.203528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:44.261927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 559/1113] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 30 00:49:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:49:44.412615 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d22fff-9e49-42c0-9725-424a1fb5773a None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:49:44.414119 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-28d22fff-9e49-42c0-9725-424a1fb5773a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:44.573342 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d22fff-9e49-42c0-9725-424a1fb5773a None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:49:44.630998 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:44.690534 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e98b7e-ac23-4646-9b2d-09ca5c3c5a8b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:49:44.691137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e98b7e-ac23-4646-9b2d-09ca5c3c5a8b service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:44.691411 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e98b7e-ac23-4646-9b2d-09ca5c3c5a8b service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:44.691680 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e98b7e-ac23-4646-9b2d-09ca5c3c5a8b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:44.693299 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-82e98b7e-ac23-4646-9b2d-09ca5c3c5a8b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:44.790465 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e98b7e-ac23-4646-9b2d-09ca5c3c5a8b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:49:44.792118 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e98b7e-ac23-4646-9b2d-09ca5c3c5a8b service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:49:44.811334 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 555/1114] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:49:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:49:45.865498 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a455b3-b0f2-4aa5-a179-9078522262f8 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:45.866073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a455b3-b0f2-4aa5-a179-9078522262f8 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:45.866073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a455b3-b0f2-4aa5-a179-9078522262f8 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:45.867372 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 560/1115] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 30 00:49:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:49:45.873526 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44169a13-d40a-4750-a7eb-32492515bceb None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:45.873964 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44169a13-d40a-4750-a7eb-32492515bceb None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:45.874079 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44169a13-d40a-4750-a7eb-32492515bceb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:45.971565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-44169a13-d40a-4750-a7eb-32492515bceb None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:45.972791 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-44169a13-d40a-4750-a7eb-32492515bceb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:45.974932 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:46.074848 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 556/1116] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 30 00:49:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:49:46.225817 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a455b3-b0f2-4aa5-a179-9078522262f8 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:49:46.227239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-14a455b3-b0f2-4aa5-a179-9078522262f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:46.385235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a455b3-b0f2-4aa5-a179-9078522262f8 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:49:46.413236 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:46.505078 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411fa2bb-8655-466a-a038-161775478d29 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:49:46.505691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411fa2bb-8655-466a-a038-161775478d29 service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:46.505966 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411fa2bb-8655-466a-a038-161775478d29 service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:46.506204 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411fa2bb-8655-466a-a038-161775478d29 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:46.507816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-411fa2bb-8655-466a-a038-161775478d29 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:46.598938 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411fa2bb-8655-466a-a038-161775478d29 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:49:46.600552 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411fa2bb-8655-466a-a038-161775478d29 service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:49:46.618729 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 561/1117] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:49:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:49:48.687135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf79e30-39ef-4693-88ab-8396979e8535 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:48.688039 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf79e30-39ef-4693-88ab-8396979e8535 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:48.688039 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf79e30-39ef-4693-88ab-8396979e8535 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:48.689861 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 557/1118] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 30 00:49:48 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:49:48.695527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e0591b-e2d4-4b39-a918-8c921e7a1b1c None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:48.695992 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e0591b-e2d4-4b39-a918-8c921e7a1b1c None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:48.696096 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e0591b-e2d4-4b39-a918-8c921e7a1b1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:48.738603 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:48.767061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-27e0591b-e2d4-4b39-a918-8c921e7a1b1c None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:48.768402 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-27e0591b-e2d4-4b39-a918-8c921e7a1b1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:48.871909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 562/1119] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 30 00:49:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:49:49.155873 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf79e30-39ef-4693-88ab-8396979e8535 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:49:49.158363 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-2bf79e30-39ef-4693-88ab-8396979e8535 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:49.272371 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf79e30-39ef-4693-88ab-8396979e8535 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:49:49.307561 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:49.357816 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02409497-5d67-463d-819a-a0fc82bc4556 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:49:49.358408 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02409497-5d67-463d-819a-a0fc82bc4556 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:49.358702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02409497-5d67-463d-819a-a0fc82bc4556 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:49.358967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02409497-5d67-463d-819a-a0fc82bc4556 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:49.360609 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-02409497-5d67-463d-819a-a0fc82bc4556 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:49.479006 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02409497-5d67-463d-819a-a0fc82bc4556 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:49:49.480393 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02409497-5d67-463d-819a-a0fc82bc4556 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:49:49.500811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 558/1120] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:49:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:49:51.836626 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cb0211-27e0-42e4-a0d2-62d7a4d29953 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:51.837187 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cb0211-27e0-42e4-a0d2-62d7a4d29953 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:51.837187 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cb0211-27e0-42e4-a0d2-62d7a4d29953 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:51.838495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 563/1121] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 30 00:49:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:49:51.845131 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143ff3ef-5533-450e-a293-210c9046b3ce None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:51.845877 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143ff3ef-5533-450e-a293-210c9046b3ce None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:51.846070 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143ff3ef-5533-450e-a293-210c9046b3ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:51.921886 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-143ff3ef-5533-450e-a293-210c9046b3ce None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:51.923077 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-143ff3ef-5533-450e-a293-210c9046b3ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:51.935994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:52.023481 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 559/1122] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 30 00:49:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:49:52.318494 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cb0211-27e0-42e4-a0d2-62d7a4d29953 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:49:52.320949 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-31cb0211-27e0-42e4-a0d2-62d7a4d29953 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:52.434269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cb0211-27e0-42e4-a0d2-62d7a4d29953 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:49:52.447577 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:52.520891 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc21666-2253-42fb-9f37-f7d9a149a3e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:49:52.521495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc21666-2253-42fb-9f37-f7d9a149a3e4 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:52.521767 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc21666-2253-42fb-9f37-f7d9a149a3e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:52.522032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc21666-2253-42fb-9f37-f7d9a149a3e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:52.523625 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-cfc21666-2253-42fb-9f37-f7d9a149a3e4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:52.640349 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc21666-2253-42fb-9f37-f7d9a149a3e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:49:52.641780 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc21666-2253-42fb-9f37-f7d9a149a3e4 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:49:52.659949 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 564/1123] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:49:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:49:55.253834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd58a1f3-9650-48cd-b966-3736130d40b3 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:55.254448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd58a1f3-9650-48cd-b966-3736130d40b3 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:55.254448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd58a1f3-9650-48cd-b966-3736130d40b3 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:55.255701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 560/1124] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 30 00:49:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:49:55.262110 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af530b7-fad6-43c3-8150-2e820ab1566d None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:55.262827 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af530b7-fad6-43c3-8150-2e820ab1566d None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:55.262827 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af530b7-fad6-43c3-8150-2e820ab1566d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:55.313414 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:55.336390 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-5af530b7-fad6-43c3-8150-2e820ab1566d None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:55.337641 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-5af530b7-fad6-43c3-8150-2e820ab1566d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:55.444905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 565/1125] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 30 00:49:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:49:55.739960 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd58a1f3-9650-48cd-b966-3736130d40b3 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:49:55.741614 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-bd58a1f3-9650-48cd-b966-3736130d40b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:55.855875 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd58a1f3-9650-48cd-b966-3736130d40b3 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:49:55.897295 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:55.932866 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fdb35f-0d5f-4c9e-a5b1-337479de97af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:49:55.933447 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fdb35f-0d5f-4c9e-a5b1-337479de97af service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:55.933722 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fdb35f-0d5f-4c9e-a5b1-337479de97af service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:55.933983 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fdb35f-0d5f-4c9e-a5b1-337479de97af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:55.935612 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-12fdb35f-0d5f-4c9e-a5b1-337479de97af service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:56.049875 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fdb35f-0d5f-4c9e-a5b1-337479de97af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:49:56.051272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fdb35f-0d5f-4c9e-a5b1-337479de97af service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:49:56.069363 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 561/1126] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:49:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 00:49:58.397690 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6c6b4b-891f-425f-b1bc-76ea6dce08cb None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:58.398221 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6c6b4b-891f-425f-b1bc-76ea6dce08cb None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:58.398221 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6c6b4b-891f-425f-b1bc-76ea6dce08cb None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:58.399563 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 566/1127] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 30 00:49:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:49:58.405837 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5ab7e6-37f5-4cf8-9a16-8205780a7e44 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:58.406552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5ab7e6-37f5-4cf8-9a16-8205780a7e44 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:58.406654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5ab7e6-37f5-4cf8-9a16-8205780a7e44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:58.483439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-5e5ab7e6-37f5-4cf8-9a16-8205780a7e44 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:49:58.484690 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-5e5ab7e6-37f5-4cf8-9a16-8205780a7e44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:58.507133 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:58.588664 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 562/1128] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 30 00:49:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:49:58.874388 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6c6b4b-891f-425f-b1bc-76ea6dce08cb None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:49:58.877159 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-4d6c6b4b-891f-425f-b1bc-76ea6dce08cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:58.991442 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6c6b4b-891f-425f-b1bc-76ea6dce08cb None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:49:59.020603 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:49:59.076328 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d06399e-8be2-4e82-bfc6-1ea821f7bb6a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:49:59.076971 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d06399e-8be2-4e82-bfc6-1ea821f7bb6a service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:49:59.077240 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d06399e-8be2-4e82-bfc6-1ea821f7bb6a service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:49:59.077517 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d06399e-8be2-4e82-bfc6-1ea821f7bb6a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:49:59.079414 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-9d06399e-8be2-4e82-bfc6-1ea821f7bb6a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:49:59.197045 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d06399e-8be2-4e82-bfc6-1ea821f7bb6a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:49:59.198402 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d06399e-8be2-4e82-bfc6-1ea821f7bb6a service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:49:59.218724 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 567/1129] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:49:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:50:01.437767 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7414c609-5323-497d-a338-10d4c7d474eb None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:01.438369 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7414c609-5323-497d-a338-10d4c7d474eb None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:01.438369 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7414c609-5323-497d-a338-10d4c7d474eb None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:01.440434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 563/1130] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 30 00:50:01 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:50:01.446959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288ee060-a1f4-4b70-b9c4-340a202fa95e None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:01.447423 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288ee060-a1f4-4b70-b9c4-340a202fa95e None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:01.447518 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288ee060-a1f4-4b70-b9c4-340a202fa95e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:01.515066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:01.518215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-288ee060-a1f4-4b70-b9c4-340a202fa95e None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:50:01.520250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-288ee060-a1f4-4b70-b9c4-340a202fa95e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:01.632459 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 568/1131] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 30 00:50:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:50:01.925742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7414c609-5323-497d-a338-10d4c7d474eb None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:50:01.927426 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-7414c609-5323-497d-a338-10d4c7d474eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:02.048203 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7414c609-5323-497d-a338-10d4c7d474eb None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:50:02.105821 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:02.132030 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65187aaa-d8fc-4f95-86d0-5bbf4126637d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:50:02.132634 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65187aaa-d8fc-4f95-86d0-5bbf4126637d service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:02.132909 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65187aaa-d8fc-4f95-86d0-5bbf4126637d service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:02.133176 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65187aaa-d8fc-4f95-86d0-5bbf4126637d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:02.134762 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-65187aaa-d8fc-4f95-86d0-5bbf4126637d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:02.260620 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65187aaa-d8fc-4f95-86d0-5bbf4126637d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:50:02.262090 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65187aaa-d8fc-4f95-86d0-5bbf4126637d service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:50:02.280334 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 564/1132] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:50:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:50:04.550309 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cd8635-ca4a-4b32-b857-4bf82cd029f1 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:04.551143 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cd8635-ca4a-4b32-b857-4bf82cd029f1 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:04.551143 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cd8635-ca4a-4b32-b857-4bf82cd029f1 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:04.553216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 569/1133] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 30 00:50:04 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:50:04.560775 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e35205-bd94-4673-85a6-edd10d64da8b None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:04.561220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e35205-bd94-4673-85a6-edd10d64da8b None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:04.561322 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e35205-bd94-4673-85a6-edd10d64da8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:04.641090 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-b7e35205-bd94-4673-85a6-edd10d64da8b None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:50:04.642199 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b7e35205-bd94-4673-85a6-edd10d64da8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:04.672582 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:04.753800 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 565/1134] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 30 00:50:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:50:05.037015 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14cd8635-ca4a-4b32-b857-4bf82cd029f1 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:50:05.039575 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-14cd8635-ca4a-4b32-b857-4bf82cd029f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:05.151369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14cd8635-ca4a-4b32-b857-4bf82cd029f1 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:50:05.186181 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:05.234557 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f054b28-696c-47e8-9c28-3f86c681d114 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:50:05.235191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f054b28-696c-47e8-9c28-3f86c681d114 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:05.235460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f054b28-696c-47e8-9c28-3f86c681d114 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:05.235756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f054b28-696c-47e8-9c28-3f86c681d114 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:05.237394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-5f054b28-696c-47e8-9c28-3f86c681d114 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:05.354558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f054b28-696c-47e8-9c28-3f86c681d114 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:50:05.356099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f054b28-696c-47e8-9c28-3f86c681d114 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:50:05.376300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 570/1135] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:50:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:50:07.637637 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49d3d78-fe78-4a62-ae5b-18a4dc82062a None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:07.638502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49d3d78-fe78-4a62-ae5b-18a4dc82062a None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:07.638502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49d3d78-fe78-4a62-ae5b-18a4dc82062a None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:07.640563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 566/1136] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 30 00:50:07 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:50:07.647050 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0411ef-4709-4e2b-8bf1-e04b7ff5d885 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:07.647546 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0411ef-4709-4e2b-8bf1-e04b7ff5d885 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:07.647652 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0411ef-4709-4e2b-8bf1-e04b7ff5d885 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:07.718866 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-cc0411ef-4709-4e2b-8bf1-e04b7ff5d885 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:50:07.720194 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-cc0411ef-4709-4e2b-8bf1-e04b7ff5d885 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:07.728807 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:07.821783 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 571/1137] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 30 00:50:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:50:08.107503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49d3d78-fe78-4a62-ae5b-18a4dc82062a None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:50:08.109098 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a49d3d78-fe78-4a62-ae5b-18a4dc82062a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:08.219398 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49d3d78-fe78-4a62-ae5b-18a4dc82062a None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:50:08.276719 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:08.296572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b09dc53-569f-46c0-98fe-d214d67b10dc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:50:08.297181 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b09dc53-569f-46c0-98fe-d214d67b10dc service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:08.297514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b09dc53-569f-46c0-98fe-d214d67b10dc service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:08.297782 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b09dc53-569f-46c0-98fe-d214d67b10dc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:08.299405 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-0b09dc53-569f-46c0-98fe-d214d67b10dc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:08.421868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b09dc53-569f-46c0-98fe-d214d67b10dc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:50:08.423219 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b09dc53-569f-46c0-98fe-d214d67b10dc service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:50:08.442107 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 567/1138] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:50:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 00:50:10.657703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba386bec-fe96-490d-af64-02902f6511c3 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:10.658328 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba386bec-fe96-490d-af64-02902f6511c3 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:10.658328 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba386bec-fe96-490d-af64-02902f6511c3 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:10.659545 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 572/1139] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 30 00:50:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:50:10.666072 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab503989-7c44-4a4f-a8c4-867576aec99e None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:10.666854 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab503989-7c44-4a4f-a8c4-867576aec99e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:10.667076 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab503989-7c44-4a4f-a8c4-867576aec99e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:10.749225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-ab503989-7c44-4a4f-a8c4-867576aec99e None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:50:10.750460 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-ab503989-7c44-4a4f-a8c4-867576aec99e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:10.789839 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:10.853284 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 568/1140] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 30 00:50:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:50:11.145600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba386bec-fe96-490d-af64-02902f6511c3 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:50:11.147168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ba386bec-fe96-490d-af64-02902f6511c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:11.252638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba386bec-fe96-490d-af64-02902f6511c3 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:50:11.293419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:11.333401 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c227ea0-9bc7-47d0-aacf-392ae3938c5d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:50:11.333930 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c227ea0-9bc7-47d0-aacf-392ae3938c5d service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:11.334203 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c227ea0-9bc7-47d0-aacf-392ae3938c5d service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:11.334473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c227ea0-9bc7-47d0-aacf-392ae3938c5d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:11.336149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-0c227ea0-9bc7-47d0-aacf-392ae3938c5d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:11.451093 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c227ea0-9bc7-47d0-aacf-392ae3938c5d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:50:11.452562 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c227ea0-9bc7-47d0-aacf-392ae3938c5d service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:50:11.471061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 573/1141] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:50:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:50:13.639560 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d517d6-0141-4983-bb7a-3a8f8ce32b34 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:13.640164 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d517d6-0141-4983-bb7a-3a8f8ce32b34 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:13.640164 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d517d6-0141-4983-bb7a-3a8f8ce32b34 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:13.641395 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 569/1142] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 30 00:50:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:50:13.647026 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03425b32-f745-46fa-804f-66342c5f2567 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:13.647481 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03425b32-f745-46fa-804f-66342c5f2567 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:13.647601 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03425b32-f745-46fa-804f-66342c5f2567 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:13.719969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-03425b32-f745-46fa-804f-66342c5f2567 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:50:13.721152 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-03425b32-f745-46fa-804f-66342c5f2567 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:13.739101 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:13.821562 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 574/1143] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 30 00:50:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:50:14.105264 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d517d6-0141-4983-bb7a-3a8f8ce32b34 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:50:14.106818 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-f9d517d6-0141-4983-bb7a-3a8f8ce32b34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:14.233093 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d517d6-0141-4983-bb7a-3a8f8ce32b34 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:50:14.310846 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:14.318405 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca80fb5-a6e2-47af-a446-482c1cfc5807 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:50:14.319047 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca80fb5-a6e2-47af-a446-482c1cfc5807 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:14.319321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca80fb5-a6e2-47af-a446-482c1cfc5807 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:14.319587 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca80fb5-a6e2-47af-a446-482c1cfc5807 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:14.321268 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-9ca80fb5-a6e2-47af-a446-482c1cfc5807 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:14.436783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca80fb5-a6e2-47af-a446-482c1cfc5807 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:50:14.438246 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca80fb5-a6e2-47af-a446-482c1cfc5807 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:50:14.456657 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 570/1144] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:50:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:50:16.636988 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b887ba-de23-45f2-a84c-2d0ed312a5f7 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:16.637539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b887ba-de23-45f2-a84c-2d0ed312a5f7 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:16.637539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b887ba-de23-45f2-a84c-2d0ed312a5f7 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:16.638805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 575/1145] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 30 00:50:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:50:16.647844 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93257168-787d-4e00-94c5-9bcbfa65faed None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:16.648583 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93257168-787d-4e00-94c5-9bcbfa65faed None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:16.648760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93257168-787d-4e00-94c5-9bcbfa65faed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:16.724161 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-93257168-787d-4e00-94c5-9bcbfa65faed None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:50:16.725362 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-93257168-787d-4e00-94c5-9bcbfa65faed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:16.776437 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:16.832304 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 571/1146] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 30 00:50:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:50:17.120845 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b887ba-de23-45f2-a84c-2d0ed312a5f7 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:50:17.122421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e3b887ba-de23-45f2-a84c-2d0ed312a5f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:17.235036 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b887ba-de23-45f2-a84c-2d0ed312a5f7 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:50:17.286939 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:17.316802 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0911230-7428-4f80-8231-a5bb00f5ab96 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:50:17.317414 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0911230-7428-4f80-8231-a5bb00f5ab96 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:17.317689 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0911230-7428-4f80-8231-a5bb00f5ab96 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:17.317958 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0911230-7428-4f80-8231-a5bb00f5ab96 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:17.319554 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c0911230-7428-4f80-8231-a5bb00f5ab96 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:17.436991 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0911230-7428-4f80-8231-a5bb00f5ab96 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:50:17.438427 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0911230-7428-4f80-8231-a5bb00f5ab96 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:50:17.458504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 576/1147] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:50:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 00:50:26.830728 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=22 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:27.455781 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=66 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:37.434533 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59699e55-8458-4719-88c1-ee2a31d7f251 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:37.435412 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59699e55-8458-4719-88c1-ee2a31d7f251 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:37.435412 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59699e55-8458-4719-88c1-ee2a31d7f251 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:37.437659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 572/1148] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:50:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:50:37.445806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1534e146-51a0-4163-9a31-e7a83478bf58 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:37.446261 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1534e146-51a0-4163-9a31-e7a83478bf58 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:37.446367 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1534e146-51a0-4163-9a31-e7a83478bf58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:37.515396 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-1534e146-51a0-4163-9a31-e7a83478bf58 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:50:37.516616 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-1534e146-51a0-4163-9a31-e7a83478bf58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:37.618066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 577/1149] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:50:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:50:37.924184 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5b135d-ea4a-4689-9646-44ce5265df19 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:37.924464 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5b135d-ea4a-4689-9646-44ce5265df19 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:37.924727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5b135d-ea4a-4689-9646-44ce5265df19 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:37.926061 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 573/1150] 10.222.0.16 () {54 vars in 962 bytes} [Sun Oct 30 00:50:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:50:37.932535 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5b0fa8-4b4f-46f0-8962-f4b4b4c84b81 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:37.932941 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5b0fa8-4b4f-46f0-8962-f4b4b4c84b81 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:37.933331 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5b0fa8-4b4f-46f0-8962-f4b4b4c84b81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:38.022330 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-db5b0fa8-4b4f-46f0-8962-f4b4b4c84b81 None None] MFA Rules not processed for user `807b123652d245269336fcffaf36478e`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:50:38.023508 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-db5b0fa8-4b4f-46f0-8962-f4b4b4c84b81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:38.045728 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:38.112221 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 578/1151] 10.222.0.16 () {58 vars in 1059 bytes} [Sun Oct 30 00:50:37 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:50:38.120825 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5b135d-ea4a-4689-9646-44ce5265df19 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:50:38.122348 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-4c5b135d-ea4a-4689-9646-44ce5265df19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:38.228572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5b135d-ea4a-4689-9646-44ce5265df19 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:50:38.305108 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7704da2-048b-46f1-81c0-1b81272838de service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '807b123652d245269336fcffaf36478e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:50:38.305731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7704da2-048b-46f1-81c0-1b81272838de service glance] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:38.306066 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7704da2-048b-46f1-81c0-1b81272838de service glance] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:38.306367 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7704da2-048b-46f1-81c0-1b81272838de service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:38.307958 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d7704da2-048b-46f1-81c0-1b81272838de service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:38.362076 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:38.418786 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7704da2-048b-46f1-81c0-1b81272838de service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:50:38.420254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7704da2-048b-46f1-81c0-1b81272838de service glance] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:50:38.440779 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 574/1152] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:50:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:50:40.792654 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5713c733-0214-42e0-8d91-b576cfbf2fe6 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:40.793233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5713c733-0214-42e0-8d91-b576cfbf2fe6 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:40.793233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5713c733-0214-42e0-8d91-b576cfbf2fe6 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:40.795257 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 579/1153] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:50:40 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:50:40.802473 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6d1908-f036-42a1-ab66-74e9e8d13d54 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:40.803103 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6d1908-f036-42a1-ab66-74e9e8d13d54 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:40.803103 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6d1908-f036-42a1-ab66-74e9e8d13d54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:40.874066 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-6f6d1908-f036-42a1-ab66-74e9e8d13d54 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:50:40.875267 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-6f6d1908-f036-42a1-ab66-74e9e8d13d54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:40.981337 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 575/1154] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:50:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:50:41.256813 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5713c733-0214-42e0-8d91-b576cfbf2fe6 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:50:41.258357 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-5713c733-0214-42e0-8d91-b576cfbf2fe6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:41.399144 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5713c733-0214-42e0-8d91-b576cfbf2fe6 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:50:41.461198 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:41.509759 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da36d2be-885a-4426-bead-8699679c751c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:50:41.510367 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da36d2be-885a-4426-bead-8699679c751c admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:41.510644 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da36d2be-885a-4426-bead-8699679c751c admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:41.510908 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da36d2be-885a-4426-bead-8699679c751c admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:41.511534 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da36d2be-885a-4426-bead-8699679c751c admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:50:41.512682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: WARNING py.warnings [None req-da36d2be-885a-4426-bead-8699679c751c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 30 00:50:41.512682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: warnings.warn(msg) Oct 30 00:50:41.512682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: Oct 30 00:50:41.513890 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da36d2be-885a-4426-bead-8699679c751c admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:50:41.533532 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 580/1155] 10.222.0.16 () {56 vars in 1108 bytes} [Sun Oct 30 00:50:41 2022] GET /identity/v3/projects => generated 2553 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 00:50:43.807701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7c3796-f51c-41bf-bc1d-8dc07f285ab2 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:43.808372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7c3796-f51c-41bf-bc1d-8dc07f285ab2 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:43.808372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7c3796-f51c-41bf-bc1d-8dc07f285ab2 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:43.809771 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 576/1156] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:50:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:50:43.815778 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36dab95-33ce-49ed-b85f-bccfbea3d7e2 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:43.816221 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36dab95-33ce-49ed-b85f-bccfbea3d7e2 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:43.816322 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36dab95-33ce-49ed-b85f-bccfbea3d7e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:43.882895 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-a36dab95-33ce-49ed-b85f-bccfbea3d7e2 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:50:43.884610 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-a36dab95-33ce-49ed-b85f-bccfbea3d7e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:43.989295 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 581/1157] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:50:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:50:44.284252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7c3796-f51c-41bf-bc1d-8dc07f285ab2 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:50:44.285949 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-fa7c3796-f51c-41bf-bc1d-8dc07f285ab2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:44.296090 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:44.449766 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7c3796-f51c-41bf-bc1d-8dc07f285ab2 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:50:44.545569 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:44.583944 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb74891-fbf1-4b9f-9100-ac9a9f560c5e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:50:44.584751 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb74891-fbf1-4b9f-9100-ac9a9f560c5e service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:44.585045 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb74891-fbf1-4b9f-9100-ac9a9f560c5e service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:44.585308 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb74891-fbf1-4b9f-9100-ac9a9f560c5e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:44.588148 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-8bb74891-fbf1-4b9f-9100-ac9a9f560c5e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:44.688361 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb74891-fbf1-4b9f-9100-ac9a9f560c5e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:50:44.690645 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bb74891-fbf1-4b9f-9100-ac9a9f560c5e service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:50:44.710826 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 577/1158] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:50:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:50:47.051655 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5ea49e-26ae-4b79-8ac6-a5febbd6519b None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:47.052267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5ea49e-26ae-4b79-8ac6-a5febbd6519b None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:47.052267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5ea49e-26ae-4b79-8ac6-a5febbd6519b None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:47.053566 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 582/1159] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:50:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:50:47.060428 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bd7b19-ee25-4512-a1da-cfa7ee68d6bc None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:47.061048 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bd7b19-ee25-4512-a1da-cfa7ee68d6bc None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:47.061194 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bd7b19-ee25-4512-a1da-cfa7ee68d6bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:47.137311 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-b1bd7b19-ee25-4512-a1da-cfa7ee68d6bc None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:50:47.138535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b1bd7b19-ee25-4512-a1da-cfa7ee68d6bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:47.188036 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:47.244455 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 578/1160] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:50:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:50:47.526731 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5ea49e-26ae-4b79-8ac6-a5febbd6519b None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:50:47.528289 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ae5ea49e-26ae-4b79-8ac6-a5febbd6519b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:47.556435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:47.641483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5ea49e-26ae-4b79-8ac6-a5febbd6519b None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:50:47.730182 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02574e3-b3c8-40c3-8b60-c684390bff97 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:50:47.731100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02574e3-b3c8-40c3-8b60-c684390bff97 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:47.731432 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02574e3-b3c8-40c3-8b60-c684390bff97 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:47.731795 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02574e3-b3c8-40c3-8b60-c684390bff97 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:47.733380 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-d02574e3-b3c8-40c3-8b60-c684390bff97 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:47.817627 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:47.853560 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02574e3-b3c8-40c3-8b60-c684390bff97 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:50:47.854995 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02574e3-b3c8-40c3-8b60-c684390bff97 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:50:47.873247 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 583/1161] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:50:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:50:50.238947 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7cfb15-4847-466d-b3a0-c62639c08dee None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:50.239519 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7cfb15-4847-466d-b3a0-c62639c08dee None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:50.239519 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7cfb15-4847-466d-b3a0-c62639c08dee None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:50.240771 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 579/1162] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:50:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:50:50.246163 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26218c07-f84c-44f6-8413-8839ac178a22 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:50.246591 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26218c07-f84c-44f6-8413-8839ac178a22 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:50.246705 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26218c07-f84c-44f6-8413-8839ac178a22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:50.315671 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-26218c07-f84c-44f6-8413-8839ac178a22 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:50:50.316889 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-26218c07-f84c-44f6-8413-8839ac178a22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:50.417408 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 584/1163] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:50:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:50:50.712869 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7cfb15-4847-466d-b3a0-c62639c08dee None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:50:50.715393 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-1d7cfb15-4847-466d-b3a0-c62639c08dee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:50.839851 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7cfb15-4847-466d-b3a0-c62639c08dee None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:50:50.926554 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:50.928201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e7f3d4-434e-4817-a9e2-988d5264f9a1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:50:50.928850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e7f3d4-434e-4817-a9e2-988d5264f9a1 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:50.929129 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e7f3d4-434e-4817-a9e2-988d5264f9a1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:50.929682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e7f3d4-434e-4817-a9e2-988d5264f9a1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:50.931749 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-22e7f3d4-434e-4817-a9e2-988d5264f9a1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:51.047860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e7f3d4-434e-4817-a9e2-988d5264f9a1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:50:51.049340 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e7f3d4-434e-4817-a9e2-988d5264f9a1 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:50:51.069328 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 580/1164] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:50:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:50:53.328745 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60724cef-d8a0-4df7-b163-a562054c645f None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:53.329565 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60724cef-d8a0-4df7-b163-a562054c645f None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:53.329565 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60724cef-d8a0-4df7-b163-a562054c645f None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:53.331545 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 585/1165] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:50:53 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:50:53.339309 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba2ab7b-7166-446a-b1b8-762e83676de7 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:53.339777 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba2ab7b-7166-446a-b1b8-762e83676de7 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:53.339889 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba2ab7b-7166-446a-b1b8-762e83676de7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:53.409769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-cba2ab7b-7166-446a-b1b8-762e83676de7 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:50:53.410974 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-cba2ab7b-7166-446a-b1b8-762e83676de7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:53.469458 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:53.516311 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 581/1166] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:50:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:50:53.807544 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60724cef-d8a0-4df7-b163-a562054c645f None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:50:53.810049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-60724cef-d8a0-4df7-b163-a562054c645f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:53.852092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:53.922809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60724cef-d8a0-4df7-b163-a562054c645f None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:50:54.013293 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4712c60-82b4-4de5-8f24-ace41329cee3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:50:54.013894 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4712c60-82b4-4de5-8f24-ace41329cee3 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:54.014168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4712c60-82b4-4de5-8f24-ace41329cee3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:54.014435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4712c60-82b4-4de5-8f24-ace41329cee3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:54.016038 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-d4712c60-82b4-4de5-8f24-ace41329cee3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:54.102936 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:54.133731 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4712c60-82b4-4de5-8f24-ace41329cee3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:50:54.135158 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4712c60-82b4-4de5-8f24-ace41329cee3 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:50:54.157041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 586/1167] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:50:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:50:56.593027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde31e6d-2ab0-4cca-87e7-c8fe345d45f3 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:56.593613 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde31e6d-2ab0-4cca-87e7-c8fe345d45f3 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:56.593613 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde31e6d-2ab0-4cca-87e7-c8fe345d45f3 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:56.594953 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 582/1168] 10.222.0.16 () {54 vars in 927 bytes} [Sun Oct 30 00:50:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:50:56.601317 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec29e57-7d9a-4afa-a36b-d63149f2dc4b None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:56.601836 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec29e57-7d9a-4afa-a36b-d63149f2dc4b None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:56.601836 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec29e57-7d9a-4afa-a36b-d63149f2dc4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:56.675382 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-aec29e57-7d9a-4afa-a36b-d63149f2dc4b None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:50:56.676601 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-aec29e57-7d9a-4afa-a36b-d63149f2dc4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:56.774921 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 587/1169] 10.222.0.16 () {58 vars in 1024 bytes} [Sun Oct 30 00:50:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:50:57.061223 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde31e6d-2ab0-4cca-87e7-c8fe345d45f3 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:50:57.062745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-dde31e6d-2ab0-4cca-87e7-c8fe345d45f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:57.190956 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde31e6d-2ab0-4cca-87e7-c8fe345d45f3 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:50:57.267955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926029a2-86dc-434a-a5c9-ba4d964ce97f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:50:57.268554 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926029a2-86dc-434a-a5c9-ba4d964ce97f service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:50:57.268819 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926029a2-86dc-434a-a5c9-ba4d964ce97f service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:50:57.269073 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926029a2-86dc-434a-a5c9-ba4d964ce97f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:50:57.270629 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-926029a2-86dc-434a-a5c9-ba4d964ce97f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:50:57.282746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:50:57.390621 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926029a2-86dc-434a-a5c9-ba4d964ce97f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:50:57.392204 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926029a2-86dc-434a-a5c9-ba4d964ce97f service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:50:57.411404 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 583/1170] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:50:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:51:06.773754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=82 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:51:07.408786 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=50 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:52:16.576150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1f2493-d3ff-4b7b-9fa8-9f3f243bad99 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:52:16.577098 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1f2493-d3ff-4b7b-9fa8-9f3f243bad99 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:52:16.577098 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1f2493-d3ff-4b7b-9fa8-9f3f243bad99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:52:16.658025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-eb1f2493-d3ff-4b7b-9fa8-9f3f243bad99 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:52:16.659176 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-eb1f2493-d3ff-4b7b-9fa8-9f3f243bad99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:52:16.767794 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 588/1171] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:52:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:52:16.776105 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f990d74-32bf-4584-b2ab-b0225a2abcf7 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:52:16.776707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f990d74-32bf-4584-b2ab-b0225a2abcf7 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:52:16.776707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f990d74-32bf-4584-b2ab-b0225a2abcf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:52:16.864961 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-2f990d74-32bf-4584-b2ab-b0225a2abcf7 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:52:16.866204 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-2f990d74-32bf-4584-b2ab-b0225a2abcf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:52:16.966278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 584/1172] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:52:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:52:16.971033 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1f2493-d3ff-4b7b-9fa8-9f3f243bad99 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:52:16.972274 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-eb1f2493-d3ff-4b7b-9fa8-9f3f243bad99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:52:16.983759 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1f2493-d3ff-4b7b-9fa8-9f3f243bad99 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:52:17.046980 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:52:17.088173 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0c336c-ac69-4d9a-ae77-71b9fd81b6e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:52:17.088765 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0c336c-ac69-4d9a-ae77-71b9fd81b6e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:52:17.089032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0c336c-ac69-4d9a-ae77-71b9fd81b6e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:52:17.089294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0c336c-ac69-4d9a-ae77-71b9fd81b6e7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:52:17.089922 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0c336c-ac69-4d9a-ae77-71b9fd81b6e7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:52:17.090216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0c336c-ac69-4d9a-ae77-71b9fd81b6e7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:52:17.091270 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: WARNING py.warnings [None req-eb0c336c-ac69-4d9a-ae77-71b9fd81b6e7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 30 00:52:17.091270 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: warnings.warn(msg) Oct 30 00:52:17.091270 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: Oct 30 00:52:17.092198 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0c336c-ac69-4d9a-ae77-71b9fd81b6e7 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:52:17.104012 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 589/1173] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:52:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:52:17.113712 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f990d74-32bf-4584-b2ab-b0225a2abcf7 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:52:17.123504 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f990d74-32bf-4584-b2ab-b0225a2abcf7 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:52:17.189414 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:52:17.224807 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7312be06-3931-4681-9666-5e5cc2cf99ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:52:17.225331 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7312be06-3931-4681-9666-5e5cc2cf99ee admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:52:17.225601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7312be06-3931-4681-9666-5e5cc2cf99ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:52:17.225866 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7312be06-3931-4681-9666-5e5cc2cf99ee admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:52:17.226488 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7312be06-3931-4681-9666-5e5cc2cf99ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:52:17.227532 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: WARNING py.warnings [None req-7312be06-3931-4681-9666-5e5cc2cf99ee admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 30 00:52:17.227532 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: warnings.warn(msg) Oct 30 00:52:17.227532 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: Oct 30 00:52:17.228740 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7312be06-3931-4681-9666-5e5cc2cf99ee admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:52:17.289194 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 585/1174] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:52:17 2022] POST /identity/v3/projects => generated 375 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:52:17.293749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0c336c-ac69-4d9a-ae77-71b9fd81b6e7 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:52:17.295311 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-eb0c336c-ac69-4d9a-ae77-71b9fd81b6e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:52:17.331475 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0c336c-ac69-4d9a-ae77-71b9fd81b6e7 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:52:17.436676 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3513151c-cadc-4ff5-b453-4e14d27d754b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:52:17.437388 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3513151c-cadc-4ff5-b453-4e14d27d754b admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:52:17.437758 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3513151c-cadc-4ff5-b453-4e14d27d754b admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:52:17.438065 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3513151c-cadc-4ff5-b453-4e14d27d754b admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:52:17.438957 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3513151c-cadc-4ff5-b453-4e14d27d754b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:52:17.440505 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: WARNING py.warnings [None req-3513151c-cadc-4ff5-b453-4e14d27d754b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 30 00:52:17.440505 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: warnings.warn(msg) Oct 30 00:52:17.440505 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: Oct 30 00:52:17.442184 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3513151c-cadc-4ff5-b453-4e14d27d754b admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:52:17.499862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 590/1175] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:52:17 2022] POST /identity/v3/users => generated 288 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:52:17.504408 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7312be06-3931-4681-9666-5e5cc2cf99ee admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:52:17.505930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-7312be06-3931-4681-9666-5e5cc2cf99ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:52:17.580492 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7312be06-3931-4681-9666-5e5cc2cf99ee admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:52:17.666607 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:52:17.679194 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ffbf27-80ba-473c-89a5-8e8851da7d11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:52:17.679834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ffbf27-80ba-473c-89a5-8e8851da7d11 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:52:17.680130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ffbf27-80ba-473c-89a5-8e8851da7d11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:52:17.680382 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ffbf27-80ba-473c-89a5-8e8851da7d11 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:52:17.681012 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ffbf27-80ba-473c-89a5-8e8851da7d11 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:52:17.682615 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ffbf27-80ba-473c-89a5-8e8851da7d11 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:52:17.695636 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 586/1176] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:52:17 2022] GET /identity/v3/roles => generated 1420 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:52:17.699808 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3513151c-cadc-4ff5-b453-4e14d27d754b admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:52:17.701340 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3513151c-cadc-4ff5-b453-4e14d27d754b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:52:17.712803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3513151c-cadc-4ff5-b453-4e14d27d754b admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:52:17.739404 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:52:17.819212 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4a88e1-53e3-44c2-8af9-5c14713fc81f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:52:17.820087 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4a88e1-53e3-44c2-8af9-5c14713fc81f admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:52:17.820446 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4a88e1-53e3-44c2-8af9-5c14713fc81f admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:52:17.820801 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4a88e1-53e3-44c2-8af9-5c14713fc81f admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:52:17.821531 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4a88e1-53e3-44c2-8af9-5c14713fc81f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:52:17.823130 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4a88e1-53e3-44c2-8af9-5c14713fc81f admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:52:17.836563 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 591/1177] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:52:17 2022] GET /identity/v3/roles => generated 1420 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:52:17.841484 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ffbf27-80ba-473c-89a5-8e8851da7d11 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:52:17.843569 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b7ffbf27-80ba-473c-89a5-8e8851da7d11 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:52:17.881236 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ffbf27-80ba-473c-89a5-8e8851da7d11 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:52:17.979619 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0418c8ee-3305-4c84-aa67-ff6f3edb8ee4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:52:17.980306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0418c8ee-3305-4c84-aa67-ff6f3edb8ee4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:52:17.980578 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0418c8ee-3305-4c84-aa67-ff6f3edb8ee4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:52:17.980850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0418c8ee-3305-4c84-aa67-ff6f3edb8ee4 admin admin] PATH_INFO: `/v3/projects/6590821babb241dc8956a35325c38b83/users/a263499f787f4836ba6957589b4e710e/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:52:18.004436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0418c8ee-3305-4c84-aa67-ff6f3edb8ee4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6590821babb241dc8956a35325c38b83, user_id=a263499f787f4836ba6957589b4e710e, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:52:18.007107 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0418c8ee-3305-4c84-aa67-ff6f3edb8ee4 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:52:18.034280 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 587/1178] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:52:17 2022] PUT /identity/v3/projects/6590821babb241dc8956a35325c38b83/users/a263499f787f4836ba6957589b4e710e/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:52:18.045897 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2237071-ed70-4eff-8c2c-aa5221e9701b None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:52:18.046087 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2237071-ed70-4eff-8c2c-aa5221e9701b None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:52:18.046388 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2237071-ed70-4eff-8c2c-aa5221e9701b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:52:18.104164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-e2237071-ed70-4eff-8c2c-aa5221e9701b None None] MFA Rules not processed for user `a263499f787f4836ba6957589b4e710e`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:52:18.105362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e2237071-ed70-4eff-8c2c-aa5221e9701b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:52:18.186727 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 592/1179] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:52:18 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:52:18.202079 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526d3634-e382-4775-8aae-2883c05b041d None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:52:18.202540 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526d3634-e382-4775-8aae-2883c05b041d None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:52:18.202843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526d3634-e382-4775-8aae-2883c05b041d None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:52:18.204980 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 588/1180] 10.222.0.16 () {54 vars in 963 bytes} [Sun Oct 30 00:52:18 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:52:18.210227 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a12f497-0448-4afa-a2f3-bf05c4c40938 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:52:18.210473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a12f497-0448-4afa-a2f3-bf05c4c40938 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:52:18.210766 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a12f497-0448-4afa-a2f3-bf05c4c40938 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:52:18.243030 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:52:18.288754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-6a12f497-0448-4afa-a2f3-bf05c4c40938 None None] MFA Rules not processed for user `d57b1c923dff462ab9aaf238f95c6b7f`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:52:18.290088 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-6a12f497-0448-4afa-a2f3-bf05c4c40938 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:52:18.370092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 593/1181] 10.222.0.16 () {58 vars in 1060 bytes} [Sun Oct 30 00:52:18 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:52:18.380407 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526d3634-e382-4775-8aae-2883c05b041d None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:52:18.381946 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-526d3634-e382-4775-8aae-2883c05b041d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:52:18.436958 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:52:18.492505 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526d3634-e382-4775-8aae-2883c05b041d None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:52:18.573778 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015a4161-974d-4c89-bef9-d12b52dfcba2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:52:18.574379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015a4161-974d-4c89-bef9-d12b52dfcba2 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:52:18.574656 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015a4161-974d-4c89-bef9-d12b52dfcba2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:52:18.574919 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015a4161-974d-4c89-bef9-d12b52dfcba2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:52:18.576541 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-015a4161-974d-4c89-bef9-d12b52dfcba2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:52:18.681470 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-015a4161-974d-4c89-bef9-d12b52dfcba2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:52:18.682816 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-015a4161-974d-4c89-bef9-d12b52dfcba2 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:52:18.686346 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:52:18.702254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 589/1182] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:52:18 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:52:18.754367 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c8858f-5cca-46c5-891f-2e90bd1ed846 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:52:18.754822 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c8858f-5cca-46c5-891f-2e90bd1ed846 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:52:18.754934 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c8858f-5cca-46c5-891f-2e90bd1ed846 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:52:18.756257 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 594/1183] 10.222.0.16 () {52 vars in 830 bytes} [Sun Oct 30 00:52:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:52:18.760378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015a4161-974d-4c89-bef9-d12b52dfcba2 service cinder] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:52:18.761670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-015a4161-974d-4c89-bef9-d12b52dfcba2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:52:18.875950 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015a4161-974d-4c89-bef9-d12b52dfcba2 service cinder] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:52:18.972813 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6e5e9e-0ab1-4d0b-9f00-99c62da23fcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:52:18.973448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6e5e9e-0ab1-4d0b-9f00-99c62da23fcb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:52:18.973715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6e5e9e-0ab1-4d0b-9f00-99c62da23fcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:52:18.973976 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6e5e9e-0ab1-4d0b-9f00-99c62da23fcb admin admin] PATH_INFO: `/v3/users/a263499f787f4836ba6957589b4e710e` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:52:18.987179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf6e5e9e-0ab1-4d0b-9f00-99c62da23fcb admin admin] RBAC: Authorizing `identity:delete_user(user_id=a263499f787f4836ba6957589b4e710e)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:52:18.988250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: WARNING py.warnings [None req-bf6e5e9e-0ab1-4d0b-9f00-99c62da23fcb admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 30 00:52:18.988250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: warnings.warn(msg) Oct 30 00:52:18.988250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: Oct 30 00:52:18.989452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf6e5e9e-0ab1-4d0b-9f00-99c62da23fcb admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:52:18.999432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:52:19.059686 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-bf6e5e9e-0ab1-4d0b-9f00-99c62da23fcb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a263499f787f4836ba6957589b4e710e'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:52:19.069861 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-bf6e5e9e-0ab1-4d0b-9f00-99c62da23fcb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a263499f787f4836ba6957589b4e710e'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:52:19.070284 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-bf6e5e9e-0ab1-4d0b-9f00-99c62da23fcb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a263499f787f4836ba6957589b4e710e'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:52:19.084619 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-bf6e5e9e-0ab1-4d0b-9f00-99c62da23fcb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a263499f787f4836ba6957589b4e710e'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:52:19.109693 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-bf6e5e9e-0ab1-4d0b-9f00-99c62da23fcb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a263499f787f4836ba6957589b4e710e'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:52:19.116149 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 590/1184] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:52:18 2022] DELETE /identity/v3/users/a263499f787f4836ba6957589b4e710e => generated 0 bytes in 356 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 00:52:19.120737 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd854c7-b4b0-4790-aae0-a2df1c0aeb12 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:52:19.121019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd854c7-b4b0-4790-aae0-a2df1c0aeb12 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:52:19.121294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd854c7-b4b0-4790-aae0-a2df1c0aeb12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:52:19.181850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-1dd854c7-b4b0-4790-aae0-a2df1c0aeb12 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:52:19.183042 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-1dd854c7-b4b0-4790-aae0-a2df1c0aeb12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:52:19.261828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:52:19.275572 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 595/1185] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:52:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:52:19.290943 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6e5e9e-0ab1-4d0b-9f00-99c62da23fcb admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:52:19.293008 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-bf6e5e9e-0ab1-4d0b-9f00-99c62da23fcb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:52:19.409268 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6e5e9e-0ab1-4d0b-9f00-99c62da23fcb admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:52:19.470362 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:52:19.491597 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d271c4-fc15-4506-9bf3-8645fbf752f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:52:19.492239 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d271c4-fc15-4506-9bf3-8645fbf752f8 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:52:19.492522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d271c4-fc15-4506-9bf3-8645fbf752f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:52:19.492872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d271c4-fc15-4506-9bf3-8645fbf752f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:52:19.494467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-f2d271c4-fc15-4506-9bf3-8645fbf752f8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:52:19.622492 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d271c4-fc15-4506-9bf3-8645fbf752f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:52:19.623890 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d271c4-fc15-4506-9bf3-8645fbf752f8 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:52:19.643971 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 591/1186] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:52:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:52:20.002148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd854c7-b4b0-4790-aae0-a2df1c0aeb12 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:52:20.003460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-1dd854c7-b4b0-4790-aae0-a2df1c0aeb12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:52:20.015982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd854c7-b4b0-4790-aae0-a2df1c0aeb12 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:52:20.116341 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ed8f86-c239-477c-9664-bc25c7d2bd61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:52:20.116964 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ed8f86-c239-477c-9664-bc25c7d2bd61 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:52:20.117234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ed8f86-c239-477c-9664-bc25c7d2bd61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:52:20.117497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ed8f86-c239-477c-9664-bc25c7d2bd61 admin admin] PATH_INFO: `/v3/projects/6590821babb241dc8956a35325c38b83` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:52:20.127681 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ed8f86-c239-477c-9664-bc25c7d2bd61 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6590821babb241dc8956a35325c38b83)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:52:20.128766 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: WARNING py.warnings [None req-67ed8f86-c239-477c-9664-bc25c7d2bd61 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 30 00:52:20.128766 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: warnings.warn(msg) Oct 30 00:52:20.128766 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: Oct 30 00:52:20.130009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ed8f86-c239-477c-9664-bc25c7d2bd61 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:52:20.191984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-67ed8f86-c239-477c-9664-bc25c7d2bd61 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6590821babb241dc8956a35325c38b83', 'request_id': 'req-67ed8f86-c239-477c-9664-bc25c7d2bd61'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:52:20.204507 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-67ed8f86-c239-477c-9664-bc25c7d2bd61 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6590821babb241dc8956a35325c38b83', 'request_id': 'req-67ed8f86-c239-477c-9664-bc25c7d2bd61'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:52:20.211782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-67ed8f86-c239-477c-9664-bc25c7d2bd61 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6590821babb241dc8956a35325c38b83', 'request_id': 'req-67ed8f86-c239-477c-9664-bc25c7d2bd61'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:52:20.242396 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-67ed8f86-c239-477c-9664-bc25c7d2bd61 admin admin] The token cache is being invalidate because project 6590821babb241dc8956a35325c38b83 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:52:20.242775 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-67ed8f86-c239-477c-9664-bc25c7d2bd61 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:52:20.243181 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-67ed8f86-c239-477c-9664-bc25c7d2bd61 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:52:20.244666 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 596/1187] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:52:20 2022] DELETE /identity/v3/projects/6590821babb241dc8956a35325c38b83 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:52:29.641464 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=62 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:52:30.240108 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:53:50.166184 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5edd7ee-b8ff-4229-9649-c9123133ff29 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:53:50.166825 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5edd7ee-b8ff-4229-9649-c9123133ff29 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:53:50.166825 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5edd7ee-b8ff-4229-9649-c9123133ff29 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:53:50.168663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 592/1188] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 30 00:53:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:53:50.175484 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b9d2c7-9b99-4396-86b5-ceca748228de None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:53:50.175945 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b9d2c7-9b99-4396-86b5-ceca748228de None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:53:50.176116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b9d2c7-9b99-4396-86b5-ceca748228de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:53:50.263052 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-99b9d2c7-9b99-4396-86b5-ceca748228de None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:53:50.264276 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-99b9d2c7-9b99-4396-86b5-ceca748228de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:53:50.383709 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 597/1189] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 30 00:53:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:53:50.542752 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5edd7ee-b8ff-4229-9649-c9123133ff29 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:53:50.546134 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b5edd7ee-b8ff-4229-9649-c9123133ff29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:53:50.670176 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5edd7ee-b8ff-4229-9649-c9123133ff29 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:53:50.675977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62fc253-4137-4147-9ecd-507989bad7b6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:53:50.676579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62fc253-4137-4147-9ecd-507989bad7b6 service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:53:50.676854 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62fc253-4137-4147-9ecd-507989bad7b6 service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:53:50.677122 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62fc253-4137-4147-9ecd-507989bad7b6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:53:50.685460 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62fc253-4137-4147-9ecd-507989bad7b6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:53:50.687418 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62fc253-4137-4147-9ecd-507989bad7b6 service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:53:50.697139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 593/1190] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:53:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:53:51.757229 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a97a40-e92a-4ade-8bd5-8aca200a3245 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:53:51.758104 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a97a40-e92a-4ade-8bd5-8aca200a3245 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:53:51.758104 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a97a40-e92a-4ade-8bd5-8aca200a3245 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:53:51.760245 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 598/1191] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 30 00:53:51 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:53:51.765486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc61d54-28e4-46cb-9d0f-0209d395ef28 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:53:51.765937 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc61d54-28e4-46cb-9d0f-0209d395ef28 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:53:51.766049 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc61d54-28e4-46cb-9d0f-0209d395ef28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:53:51.790932 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-bdc61d54-28e4-46cb-9d0f-0209d395ef28 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:53:51.792159 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-bdc61d54-28e4-46cb-9d0f-0209d395ef28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:53:51.801285 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 594/1192] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 30 00:53:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:53:51.954107 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a97a40-e92a-4ade-8bd5-8aca200a3245 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:53:51.980360 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a97a40-e92a-4ade-8bd5-8aca200a3245 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:53:51.988230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e0e49c-f1f4-4ab1-ae51-10311dc32d41 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:53:51.988854 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e0e49c-f1f4-4ab1-ae51-10311dc32d41 service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:53:51.989128 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e0e49c-f1f4-4ab1-ae51-10311dc32d41 service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:53:51.989398 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e0e49c-f1f4-4ab1-ae51-10311dc32d41 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:53:51.998294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e0e49c-f1f4-4ab1-ae51-10311dc32d41 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:53:51.999883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e0e49c-f1f4-4ab1-ae51-10311dc32d41 service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:53:52.010032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 599/1193] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:53:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:53:52.078683 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cf3bcb-eb3c-4302-8cb1-c4dab4f2080d None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:53:52.078953 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cf3bcb-eb3c-4302-8cb1-c4dab4f2080d None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:53:52.079215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cf3bcb-eb3c-4302-8cb1-c4dab4f2080d None None] PATH_INFO: `/v3` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:53:52.080621 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 595/1194] 10.222.0.43 () {54 vars in 925 bytes} [Sun Oct 30 00:53:52 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:53:52.087393 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bb890a-631f-4b96-81f6-0436a165fa25 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:53:52.087669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bb890a-631f-4b96-81f6-0436a165fa25 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:53:52.087960 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bb890a-631f-4b96-81f6-0436a165fa25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:53:52.157404 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-58bb890a-631f-4b96-81f6-0436a165fa25 None None] MFA Rules not processed for user `63cc035057064805a95e9f12714ebade`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:53:52.158650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-58bb890a-631f-4b96-81f6-0436a165fa25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:53:52.175357 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:53:52.233170 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 600/1195] 10.222.0.43 () {58 vars in 1015 bytes} [Sun Oct 30 00:53:52 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:54:01.643647 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d6d923-3e70-417d-a3f9-7f9adeca9f10 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:54:01.644200 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d6d923-3e70-417d-a3f9-7f9adeca9f10 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:54:01.644200 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d6d923-3e70-417d-a3f9-7f9adeca9f10 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:54:01.645587 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 596/1196] 10.222.0.16 () {54 vars in 961 bytes} [Sun Oct 30 00:54:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:54:01.651511 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c387e34-47ba-4992-8293-be864c2f526b None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:54:01.652028 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c387e34-47ba-4992-8293-be864c2f526b None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:54:01.652139 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c387e34-47ba-4992-8293-be864c2f526b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:54:01.678219 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-1c387e34-47ba-4992-8293-be864c2f526b None None] MFA Rules not processed for user `63cc035057064805a95e9f12714ebade`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:54:01.679495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-1c387e34-47ba-4992-8293-be864c2f526b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:54:01.689833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 601/1197] 10.222.0.16 () {58 vars in 1058 bytes} [Sun Oct 30 00:54:01 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:54:01.698097 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d6d923-3e70-417d-a3f9-7f9adeca9f10 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:54:01.712905 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d6d923-3e70-417d-a3f9-7f9adeca9f10 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:54:01.718684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f98aef-ee0e-4543-8ce1-0cbdd7f2b460 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '63cc035057064805a95e9f12714ebade', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:54:01.719378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f98aef-ee0e-4543-8ce1-0cbdd7f2b460 service placement] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:54:01.719656 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f98aef-ee0e-4543-8ce1-0cbdd7f2b460 service placement] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:54:01.719958 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f98aef-ee0e-4543-8ce1-0cbdd7f2b460 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:54:01.728368 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f98aef-ee0e-4543-8ce1-0cbdd7f2b460 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:54:01.729942 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f98aef-ee0e-4543-8ce1-0cbdd7f2b460 service placement] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:54:01.739914 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 597/1198] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:54:01 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:54:03.507809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5591f86-ccb9-4a84-ae66-23b836c455ee None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:54:03.508394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5591f86-ccb9-4a84-ae66-23b836c455ee None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:54:03.508394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5591f86-ccb9-4a84-ae66-23b836c455ee None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:54:03.510275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 602/1199] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 30 00:54:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:54:03.515817 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb14cf6-2b65-4809-89e5-d40885370bd4 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:54:03.516254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb14cf6-2b65-4809-89e5-d40885370bd4 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:54:03.516356 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb14cf6-2b65-4809-89e5-d40885370bd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:54:03.541287 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-6bb14cf6-2b65-4809-89e5-d40885370bd4 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:54:03.542523 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-6bb14cf6-2b65-4809-89e5-d40885370bd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:54:03.551544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 598/1200] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 30 00:54:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:54:03.711363 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5591f86-ccb9-4a84-ae66-23b836c455ee None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:54:03.729731 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5591f86-ccb9-4a84-ae66-23b836c455ee None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:54:03.735775 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8188845-3810-45dd-926f-64727be1616b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:54:03.736384 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8188845-3810-45dd-926f-64727be1616b service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:54:03.736653 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8188845-3810-45dd-926f-64727be1616b service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:54:03.736914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8188845-3810-45dd-926f-64727be1616b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:54:03.745436 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8188845-3810-45dd-926f-64727be1616b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:54:03.747018 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8188845-3810-45dd-926f-64727be1616b service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:54:03.756438 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 603/1201] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:54:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:54:04.826273 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0632e99-f898-48af-98b2-796810279ac5 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:54:04.826822 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0632e99-f898-48af-98b2-796810279ac5 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:54:04.826822 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0632e99-f898-48af-98b2-796810279ac5 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:54:04.828468 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 599/1202] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 30 00:54:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:54:04.833701 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6eb8fb-3146-449a-bc80-0a750f808b73 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:54:04.834077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6eb8fb-3146-449a-bc80-0a750f808b73 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:54:04.834239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6eb8fb-3146-449a-bc80-0a750f808b73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:54:04.857446 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-fb6eb8fb-3146-449a-bc80-0a750f808b73 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:54:04.858613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-fb6eb8fb-3146-449a-bc80-0a750f808b73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:54:04.867451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 604/1203] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 30 00:54:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:54:05.024148 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0632e99-f898-48af-98b2-796810279ac5 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:54:05.041838 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0632e99-f898-48af-98b2-796810279ac5 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:54:05.047761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219140b1-3fc0-40e2-9010-6463c95d5fdf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:54:05.048366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219140b1-3fc0-40e2-9010-6463c95d5fdf service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:54:05.048635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219140b1-3fc0-40e2-9010-6463c95d5fdf service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:54:05.048907 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219140b1-3fc0-40e2-9010-6463c95d5fdf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:54:05.057495 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219140b1-3fc0-40e2-9010-6463c95d5fdf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:54:05.059080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219140b1-3fc0-40e2-9010-6463c95d5fdf service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:54:05.069167 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 600/1204] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:54:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:54:09.195180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dcc730-1ada-4f58-b167-873a9edaeda0 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:54:09.195180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dcc730-1ada-4f58-b167-873a9edaeda0 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:54:09.195180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dcc730-1ada-4f58-b167-873a9edaeda0 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:54:09.196132 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 605/1205] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 30 00:54:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:54:09.201853 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f65810-edcf-4f90-b7df-a5695ee8c80e None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:54:09.202517 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f65810-edcf-4f90-b7df-a5695ee8c80e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:54:09.202748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f65810-edcf-4f90-b7df-a5695ee8c80e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:54:09.230302 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-66f65810-edcf-4f90-b7df-a5695ee8c80e None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:54:09.231521 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-66f65810-edcf-4f90-b7df-a5695ee8c80e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:54:09.241510 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 601/1206] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 30 00:54:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:54:09.524091 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2dcc730-1ada-4f58-b167-873a9edaeda0 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:54:09.526420 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-d2dcc730-1ada-4f58-b167-873a9edaeda0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:54:09.609845 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2dcc730-1ada-4f58-b167-873a9edaeda0 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:54:09.613922 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be61ca56-e82f-48dd-81f4-1bd3a5ce383b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:54:09.614529 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be61ca56-e82f-48dd-81f4-1bd3a5ce383b service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:54:09.614801 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be61ca56-e82f-48dd-81f4-1bd3a5ce383b service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:54:09.615067 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be61ca56-e82f-48dd-81f4-1bd3a5ce383b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:54:09.631313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be61ca56-e82f-48dd-81f4-1bd3a5ce383b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:54:09.633533 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be61ca56-e82f-48dd-81f4-1bd3a5ce383b service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:54:09.646407 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 606/1207] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:54:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:54:11.757922 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98003965-3c31-4097-ad2b-954328393c6e None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:54:11.758517 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98003965-3c31-4097-ad2b-954328393c6e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:54:11.758517 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98003965-3c31-4097-ad2b-954328393c6e None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:54:11.759841 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 602/1208] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 30 00:54:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:54:11.766299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00476537-d779-4f9f-9651-14453ccb48ad None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:54:11.767115 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00476537-d779-4f9f-9651-14453ccb48ad None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:54:11.767261 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00476537-d779-4f9f-9651-14453ccb48ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:54:11.787350 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-00476537-d779-4f9f-9651-14453ccb48ad None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:54:11.788581 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-00476537-d779-4f9f-9651-14453ccb48ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:54:11.797619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 607/1209] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 30 00:54:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:54:12.086875 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98003965-3c31-4097-ad2b-954328393c6e None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:54:12.102157 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98003965-3c31-4097-ad2b-954328393c6e None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:54:12.106914 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0296af75-4f2c-4f07-886a-0f0b29aa081c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:54:12.107645 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0296af75-4f2c-4f07-886a-0f0b29aa081c service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:54:12.107988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0296af75-4f2c-4f07-886a-0f0b29aa081c service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:54:12.108253 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0296af75-4f2c-4f07-886a-0f0b29aa081c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:54:12.117293 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0296af75-4f2c-4f07-886a-0f0b29aa081c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:54:12.118715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0296af75-4f2c-4f07-886a-0f0b29aa081c service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:54:12.128558 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 603/1210] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:54:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:54:15.937005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a742738-6e2c-468e-81e0-2a53ae1910e5 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:54:15.937586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a742738-6e2c-468e-81e0-2a53ae1910e5 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:54:15.937586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a742738-6e2c-468e-81e0-2a53ae1910e5 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:54:15.939529 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 608/1211] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 30 00:54:15 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:54:15.947833 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7525ba2c-eb23-48f5-98b1-6f2b73f11b9e None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:54:15.948249 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7525ba2c-eb23-48f5-98b1-6f2b73f11b9e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:54:15.948371 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7525ba2c-eb23-48f5-98b1-6f2b73f11b9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:54:15.972082 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-7525ba2c-eb23-48f5-98b1-6f2b73f11b9e None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:54:15.973279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-7525ba2c-eb23-48f5-98b1-6f2b73f11b9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:54:15.982871 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 604/1212] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 30 00:54:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:54:16.286531 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a742738-6e2c-468e-81e0-2a53ae1910e5 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:54:16.289387 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-2a742738-6e2c-468e-81e0-2a53ae1910e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:54:16.362833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:54:16.378241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a742738-6e2c-468e-81e0-2a53ae1910e5 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:54:16.382846 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ba89da-c8ff-4fa4-a9d9-d8ea0333b6f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:54:16.383448 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ba89da-c8ff-4fa4-a9d9-d8ea0333b6f6 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:54:16.383738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ba89da-c8ff-4fa4-a9d9-d8ea0333b6f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:54:16.384029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ba89da-c8ff-4fa4-a9d9-d8ea0333b6f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:54:16.393806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ba89da-c8ff-4fa4-a9d9-d8ea0333b6f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:54:16.395248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ba89da-c8ff-4fa4-a9d9-d8ea0333b6f6 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:54:16.404588 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 609/1213] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:54:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:54:18.545102 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e62491-f76e-4ba5-9f39-4f4ada4a9490 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:54:18.546030 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e62491-f76e-4ba5-9f39-4f4ada4a9490 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:54:18.546030 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e62491-f76e-4ba5-9f39-4f4ada4a9490 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:54:18.548381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 605/1214] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 30 00:54:18 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:54:18.553803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cdf27e-82bb-4c63-9af1-47ab10ccbc9f None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:54:18.554248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cdf27e-82bb-4c63-9af1-47ab10ccbc9f None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:54:18.554353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cdf27e-82bb-4c63-9af1-47ab10ccbc9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:54:18.579951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-c5cdf27e-82bb-4c63-9af1-47ab10ccbc9f None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:54:18.581221 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c5cdf27e-82bb-4c63-9af1-47ab10ccbc9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:54:18.589953 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 610/1215] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 30 00:54:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:54:18.891502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e62491-f76e-4ba5-9f39-4f4ada4a9490 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:54:18.895110 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-35e62491-f76e-4ba5-9f39-4f4ada4a9490 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:54:18.915737 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e62491-f76e-4ba5-9f39-4f4ada4a9490 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:54:18.919893 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c13c8cb-d72c-457c-b2a6-de9fa3188b35 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:54:18.920483 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c13c8cb-d72c-457c-b2a6-de9fa3188b35 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:54:18.920755 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c13c8cb-d72c-457c-b2a6-de9fa3188b35 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:54:18.921021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c13c8cb-d72c-457c-b2a6-de9fa3188b35 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:54:18.931614 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c13c8cb-d72c-457c-b2a6-de9fa3188b35 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:54:18.933077 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c13c8cb-d72c-457c-b2a6-de9fa3188b35 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:54:18.943340 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 606/1216] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:54:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:54:28.570889 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=11 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:54:28.938761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:43.502910 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1e45cc-9589-4b9f-9ccd-c63f02df042c None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:43.504236 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1e45cc-9589-4b9f-9ccd-c63f02df042c None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:43.504236 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1e45cc-9589-4b9f-9ccd-c63f02df042c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:43.541059 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-5b1e45cc-9589-4b9f-9ccd-c63f02df042c None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:43.542681 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-5b1e45cc-9589-4b9f-9ccd-c63f02df042c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:43.553646 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 611/1217] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:43.562934 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763a46ca-5d6b-43db-87fd-37916e6020e7 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:43.563553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763a46ca-5d6b-43db-87fd-37916e6020e7 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:43.563553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763a46ca-5d6b-43db-87fd-37916e6020e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:43.566445 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a261a7e1-221e-4ecd-a560-4f9edb7c8282 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:43.566726 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a261a7e1-221e-4ecd-a560-4f9edb7c8282 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:43.566993 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a261a7e1-221e-4ecd-a560-4f9edb7c8282 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:43.599837 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-a261a7e1-221e-4ecd-a560-4f9edb7c8282 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:43.601132 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-a261a7e1-221e-4ecd-a560-4f9edb7c8282 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:43.601281 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-763a46ca-5d6b-43db-87fd-37916e6020e7 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:43.602407 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-763a46ca-5d6b-43db-87fd-37916e6020e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:43.619287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 612/1218] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:55:43.619825 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 607/1219] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:43.624922 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a261a7e1-221e-4ecd-a560-4f9edb7c8282 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:43.629776 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa39a00e-f88f-46cc-96e8-164078cc151b None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:43.630048 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa39a00e-f88f-46cc-96e8-164078cc151b None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:43.630323 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa39a00e-f88f-46cc-96e8-164078cc151b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:43.634968 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a261a7e1-221e-4ecd-a560-4f9edb7c8282 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:43.650886 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b12bae-9a20-488e-be97-b38fc5337297 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:43.651528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b12bae-9a20-488e-be97-b38fc5337297 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:43.651827 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b12bae-9a20-488e-be97-b38fc5337297 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:43.652097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b12bae-9a20-488e-be97-b38fc5337297 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:43.652746 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b12bae-9a20-488e-be97-b38fc5337297 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:55:43.653043 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b12bae-9a20-488e-be97-b38fc5337297 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:43.656001 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b12bae-9a20-488e-be97-b38fc5337297 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:43.671802 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-fa39a00e-f88f-46cc-96e8-164078cc151b None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:43.673026 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-fa39a00e-f88f-46cc-96e8-164078cc151b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:43.681955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 613/1220] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:55:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:55:43.690568 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b12bae-9a20-488e-be97-b38fc5337297 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:43.696405 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 608/1221] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:55:43.701355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa39a00e-f88f-46cc-96e8-164078cc151b None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:43.703092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b12bae-9a20-488e-be97-b38fc5337297 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:43.711421 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa39a00e-f88f-46cc-96e8-164078cc151b None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:43.720793 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf08cc9-b79a-40d5-82af-db54dbd6c9d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:43.721524 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf08cc9-b79a-40d5-82af-db54dbd6c9d6 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:43.722077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf08cc9-b79a-40d5-82af-db54dbd6c9d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:43.722439 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf08cc9-b79a-40d5-82af-db54dbd6c9d6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:43.723094 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf08cc9-b79a-40d5-82af-db54dbd6c9d6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:43.724179 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: WARNING py.warnings [None req-5bf08cc9-b79a-40d5-82af-db54dbd6c9d6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 30 00:55:43.724179 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: warnings.warn(msg) Oct 30 00:55:43.724179 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: Oct 30 00:55:43.725521 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf08cc9-b79a-40d5-82af-db54dbd6c9d6 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:43.733647 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c443a10-31a3-4ea1-8e4e-9967a77c17fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:43.733915 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c443a10-31a3-4ea1-8e4e-9967a77c17fd admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:43.734192 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c443a10-31a3-4ea1-8e4e-9967a77c17fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:43.734460 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c443a10-31a3-4ea1-8e4e-9967a77c17fd admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:43.735129 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c443a10-31a3-4ea1-8e4e-9967a77c17fd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:55:43.735428 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c443a10-31a3-4ea1-8e4e-9967a77c17fd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:43.736713 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: WARNING py.warnings [None req-7c443a10-31a3-4ea1-8e4e-9967a77c17fd admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 30 00:55:43.736713 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: warnings.warn(msg) Oct 30 00:55:43.736713 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: Oct 30 00:55:43.737621 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c443a10-31a3-4ea1-8e4e-9967a77c17fd admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:43.770493 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 609/1222] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:55:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:55:43.772004 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a93034e-598a-4138-98bc-da1fd73305a3 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:43.772306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a93034e-598a-4138-98bc-da1fd73305a3 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:43.772575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a93034e-598a-4138-98bc-da1fd73305a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:43.794054 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 614/1223] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:55:43 2022] POST /identity/v3/projects => generated 383 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:43.795139 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf08cc9-b79a-40d5-82af-db54dbd6c9d6 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:43.808895 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-2a93034e-598a-4138-98bc-da1fd73305a3 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:43.810580 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-2a93034e-598a-4138-98bc-da1fd73305a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:43.909670 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf08cc9-b79a-40d5-82af-db54dbd6c9d6 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:43.928257 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5286fc75-4bdf-42e3-825b-f298da6f20d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:43.928894 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5286fc75-4bdf-42e3-825b-f298da6f20d0 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:43.929168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5286fc75-4bdf-42e3-825b-f298da6f20d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:43.929436 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5286fc75-4bdf-42e3-825b-f298da6f20d0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:43.930073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5286fc75-4bdf-42e3-825b-f298da6f20d0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:43.936803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5286fc75-4bdf-42e3-825b-f298da6f20d0 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:43.940794 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 610/1224] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:43.941806 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a93034e-598a-4138-98bc-da1fd73305a3 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:43.962890 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a93034e-598a-4138-98bc-da1fd73305a3 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:43.976724 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcefc96-6b20-4531-9698-1282d8031fc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:43.977473 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcefc96-6b20-4531-9698-1282d8031fc6 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:43.977760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcefc96-6b20-4531-9698-1282d8031fc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:43.978034 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcefc96-6b20-4531-9698-1282d8031fc6 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:43.978911 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bcefc96-6b20-4531-9698-1282d8031fc6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:43.980025 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: WARNING py.warnings [None req-0bcefc96-6b20-4531-9698-1282d8031fc6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 30 00:55:43.980025 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: warnings.warn(msg) Oct 30 00:55:43.980025 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: Oct 30 00:55:43.981871 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bcefc96-6b20-4531-9698-1282d8031fc6 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:43.990996 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 615/1225] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:55:43 2022] POST /identity/v3/projects => generated 375 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:43.992559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50c8c88-a86c-47bd-a67b-2485cac5a693 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:43.992889 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50c8c88-a86c-47bd-a67b-2485cac5a693 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:43.993166 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50c8c88-a86c-47bd-a67b-2485cac5a693 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.016350 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 611/1226] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:55:43 2022] POST /identity/v3/users => generated 292 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:44.018317 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cc0051-2fe9-40b7-aea2-7f18f340d32d None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.018610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cc0051-2fe9-40b7-aea2-7f18f340d32d None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.018877 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cc0051-2fe9-40b7-aea2-7f18f340d32d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.022536 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-e50c8c88-a86c-47bd-a67b-2485cac5a693 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:44.023957 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e50c8c88-a86c-47bd-a67b-2485cac5a693 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:44.039604 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-57cc0051-2fe9-40b7-aea2-7f18f340d32d None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:44.040817 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-57cc0051-2fe9-40b7-aea2-7f18f340d32d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:44.090556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 616/1227] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:44.092014 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4a69b5-18c6-4c54-abc6-a904768b88fa None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.092431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4a69b5-18c6-4c54-abc6-a904768b88fa None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.092714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4a69b5-18c6-4c54-abc6-a904768b88fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.103745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 612/1228] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:44.105151 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5b8b99-1bd9-4e17-9ca8-ddd7b17aed74 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.105532 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5b8b99-1bd9-4e17-9ca8-ddd7b17aed74 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.105940 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5b8b99-1bd9-4e17-9ca8-ddd7b17aed74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.113214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-fc4a69b5-18c6-4c54-abc6-a904768b88fa None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:44.114450 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-fc4a69b5-18c6-4c54-abc6-a904768b88fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:44.126311 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 617/1229] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:44.130225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4a69b5-18c6-4c54-abc6-a904768b88fa None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:44.132708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-4e5b8b99-1bd9-4e17-9ca8-ddd7b17aed74 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:44.134445 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-4e5b8b99-1bd9-4e17-9ca8-ddd7b17aed74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:44.139178 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4a69b5-18c6-4c54-abc6-a904768b88fa None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:44.144766 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 613/1230] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:44.145350 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6139e914-196e-43c0-ad19-449a567f0211 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:44.146190 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6139e914-196e-43c0-ad19-449a567f0211 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.146372 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6139e914-196e-43c0-ad19-449a567f0211 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.146615 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe66bb8-2b54-4596-ab55-74e0eacafc15 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.146744 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6139e914-196e-43c0-ad19-449a567f0211 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.147532 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6139e914-196e-43c0-ad19-449a567f0211 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:44.148035 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe66bb8-2b54-4596-ab55-74e0eacafc15 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.148376 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe66bb8-2b54-4596-ab55-74e0eacafc15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.149859 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6139e914-196e-43c0-ad19-449a567f0211 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:44.170750 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-cfe66bb8-2b54-4596-ab55-74e0eacafc15 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:44.171910 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-cfe66bb8-2b54-4596-ab55-74e0eacafc15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:44.173716 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 618/1231] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:55:44 2022] POST /identity/v3/users => generated 288 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:44.174545 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6139e914-196e-43c0-ad19-449a567f0211 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:44.180466 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 614/1232] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:44.181717 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e4342b-9b9e-4039-9141-95656c7f006c None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.181876 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:44.181993 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e4342b-9b9e-4039-9141-95656c7f006c None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.182253 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e4342b-9b9e-4039-9141-95656c7f006c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.185700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6139e914-196e-43c0-ad19-449a567f0211 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:44.193464 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d301f4c3-3f47-4170-95be-78d82f1eec8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:44.194402 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d301f4c3-3f47-4170-95be-78d82f1eec8c admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.194675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d301f4c3-3f47-4170-95be-78d82f1eec8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.194941 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d301f4c3-3f47-4170-95be-78d82f1eec8c admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.195579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d301f4c3-3f47-4170-95be-78d82f1eec8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:44.197305 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d301f4c3-3f47-4170-95be-78d82f1eec8c admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:44.202728 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-86e4342b-9b9e-4039-9141-95656c7f006c None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:44.203966 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-86e4342b-9b9e-4039-9141-95656c7f006c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:44.209123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 619/1233] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:55:44 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:55:44.210348 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f7c53f-2e2a-418f-ab97-8a9f365cf824 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.210626 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f7c53f-2e2a-418f-ab97-8a9f365cf824 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.210891 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f7c53f-2e2a-418f-ab97-8a9f365cf824 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.213791 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 615/1234] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:44.215025 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6712bdc-b178-4300-9e0d-eaff656e75d0 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.215301 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6712bdc-b178-4300-9e0d-eaff656e75d0 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.215562 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6712bdc-b178-4300-9e0d-eaff656e75d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.231653 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-20f7c53f-2e2a-418f-ab97-8a9f365cf824 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:44.233006 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-20f7c53f-2e2a-418f-ab97-8a9f365cf824 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:44.237387 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-b6712bdc-b178-4300-9e0d-eaff656e75d0 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:44.238543 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b6712bdc-b178-4300-9e0d-eaff656e75d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:44.241065 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 620/1235] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:44.241896 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f7c53f-2e2a-418f-ab97-8a9f365cf824 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:44.246829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 616/1236] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:44.248033 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f564bd-597e-4bfd-8a3f-dd08200c48dd None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.248309 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f564bd-597e-4bfd-8a3f-dd08200c48dd None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.248576 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f564bd-597e-4bfd-8a3f-dd08200c48dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.251558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f7c53f-2e2a-418f-ab97-8a9f365cf824 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:44.256195 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c091467a-6ef9-4bf9-9dee-bb2d920f4163 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:44.256787 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c091467a-6ef9-4bf9-9dee-bb2d920f4163 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.257058 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c091467a-6ef9-4bf9-9dee-bb2d920f4163 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.257322 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c091467a-6ef9-4bf9-9dee-bb2d920f4163 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.257950 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c091467a-6ef9-4bf9-9dee-bb2d920f4163 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:55:44.258242 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c091467a-6ef9-4bf9-9dee-bb2d920f4163 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:44.259853 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c091467a-6ef9-4bf9-9dee-bb2d920f4163 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:44.268360 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-10f564bd-597e-4bfd-8a3f-dd08200c48dd None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:44.269539 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-10f564bd-597e-4bfd-8a3f-dd08200c48dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:44.272423 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 621/1237] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:55:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:55:44.273332 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c091467a-6ef9-4bf9-9dee-bb2d920f4163 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:44.285779 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 617/1238] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:44.286964 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5519282e-13aa-42bf-99ac-8c1d907132a8 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.287302 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5519282e-13aa-42bf-99ac-8c1d907132a8 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.287601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5519282e-13aa-42bf-99ac-8c1d907132a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.292330 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c091467a-6ef9-4bf9-9dee-bb2d920f4163 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:44.298090 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707befff-28b4-47eb-8da7-97ddb2bfc3cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:44.298706 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707befff-28b4-47eb-8da7-97ddb2bfc3cf admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.298980 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707befff-28b4-47eb-8da7-97ddb2bfc3cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.299252 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707befff-28b4-47eb-8da7-97ddb2bfc3cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.299916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707befff-28b4-47eb-8da7-97ddb2bfc3cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:44.301508 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707befff-28b4-47eb-8da7-97ddb2bfc3cf admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:44.307732 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-5519282e-13aa-42bf-99ac-8c1d907132a8 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:44.308942 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-5519282e-13aa-42bf-99ac-8c1d907132a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:44.313039 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 622/1239] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:55:44 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:55:44.313846 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707befff-28b4-47eb-8da7-97ddb2bfc3cf admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:44.318365 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 618/1240] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:44.320708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a608887c-6af4-45fd-9612-92593724461f None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.320990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a608887c-6af4-45fd-9612-92593724461f None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.321261 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a608887c-6af4-45fd-9612-92593724461f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.323607 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707befff-28b4-47eb-8da7-97ddb2bfc3cf admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:44.328088 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70dce6b-8662-4042-97bf-487b1184340c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:44.328683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70dce6b-8662-4042-97bf-487b1184340c admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.328962 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70dce6b-8662-4042-97bf-487b1184340c admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.329229 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70dce6b-8662-4042-97bf-487b1184340c admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.329845 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70dce6b-8662-4042-97bf-487b1184340c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:44.331418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70dce6b-8662-4042-97bf-487b1184340c admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:44.341521 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-a608887c-6af4-45fd-9612-92593724461f None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:44.341946 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 623/1241] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:55:44 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:55:44.342633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70dce6b-8662-4042-97bf-487b1184340c admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:44.342744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a608887c-6af4-45fd-9612-92593724461f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:44.353449 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 619/1242] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:44.355784 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70dce6b-8662-4042-97bf-487b1184340c admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:44.356621 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a608887c-6af4-45fd-9612-92593724461f None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:44.360745 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7104c43c-c207-4a6c-9f74-967119f32ea2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:44.361351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7104c43c-c207-4a6c-9f74-967119f32ea2 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.361630 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7104c43c-c207-4a6c-9f74-967119f32ea2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.361927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7104c43c-c207-4a6c-9f74-967119f32ea2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.362578 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7104c43c-c207-4a6c-9f74-967119f32ea2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:55:44.362877 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7104c43c-c207-4a6c-9f74-967119f32ea2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:44.364645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7104c43c-c207-4a6c-9f74-967119f32ea2 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:44.370329 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a608887c-6af4-45fd-9612-92593724461f None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:44.375955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cac000b-17f6-476d-8986-54a85decb524 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:44.376551 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cac000b-17f6-476d-8986-54a85decb524 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.376792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 624/1243] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:55:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:55:44.376909 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cac000b-17f6-476d-8986-54a85decb524 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.377113 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cac000b-17f6-476d-8986-54a85decb524 admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.377571 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7104c43c-c207-4a6c-9f74-967119f32ea2 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:44.377705 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cac000b-17f6-476d-8986-54a85decb524 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:55:44.377994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cac000b-17f6-476d-8986-54a85decb524 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:44.379600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cac000b-17f6-476d-8986-54a85decb524 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:44.388458 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:44.393026 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7104c43c-c207-4a6c-9f74-967119f32ea2 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:44.393941 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 620/1244] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:55:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:55:44.394806 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cac000b-17f6-476d-8986-54a85decb524 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:44.404971 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cac000b-17f6-476d-8986-54a85decb524 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:44.405125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca18dd8e-d95d-49aa-a325-42d47fc95135 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:44.405882 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca18dd8e-d95d-49aa-a325-42d47fc95135 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.406272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca18dd8e-d95d-49aa-a325-42d47fc95135 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.406678 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca18dd8e-d95d-49aa-a325-42d47fc95135 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.407311 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca18dd8e-d95d-49aa-a325-42d47fc95135 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:44.409271 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca18dd8e-d95d-49aa-a325-42d47fc95135 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:44.410024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b559eb66-b450-4740-811c-b33441c32f67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:44.410612 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b559eb66-b450-4740-811c-b33441c32f67 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.410890 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b559eb66-b450-4740-811c-b33441c32f67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.411160 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b559eb66-b450-4740-811c-b33441c32f67 admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.411838 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b559eb66-b450-4740-811c-b33441c32f67 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:55:44.412136 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b559eb66-b450-4740-811c-b33441c32f67 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:44.413749 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b559eb66-b450-4740-811c-b33441c32f67 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:44.433012 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 621/1245] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:55:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:55:44.433012 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b559eb66-b450-4740-811c-b33441c32f67 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:44.436347 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b559eb66-b450-4740-811c-b33441c32f67 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:44.441402 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4eeefea-9fe3-4c93-94ec-17969b2310aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:44.442056 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4eeefea-9fe3-4c93-94ec-17969b2310aa admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.442330 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4eeefea-9fe3-4c93-94ec-17969b2310aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.442596 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4eeefea-9fe3-4c93-94ec-17969b2310aa admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.442750 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 625/1246] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:55:44 2022] POST /identity/v3/projects => generated 385 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:44.443228 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4eeefea-9fe3-4c93-94ec-17969b2310aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:44.443650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca18dd8e-d95d-49aa-a325-42d47fc95135 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:44.445408 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4eeefea-9fe3-4c93-94ec-17969b2310aa admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:44.456969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 622/1247] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:55:44 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:55:44.457821 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4eeefea-9fe3-4c93-94ec-17969b2310aa admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:44.468226 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4eeefea-9fe3-4c93-94ec-17969b2310aa admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:44.508305 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca18dd8e-d95d-49aa-a325-42d47fc95135 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:44.513006 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f478398-5d3b-4076-be5a-848c5f969241 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:44.513646 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f478398-5d3b-4076-be5a-848c5f969241 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.513919 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f478398-5d3b-4076-be5a-848c5f969241 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.514187 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f478398-5d3b-4076-be5a-848c5f969241 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.514840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f478398-5d3b-4076-be5a-848c5f969241 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:55:44.515137 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f478398-5d3b-4076-be5a-848c5f969241 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:44.516794 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f478398-5d3b-4076-be5a-848c5f969241 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:44.526043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e70b16-7c88-4cbb-b90f-ba8b575f987d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:44.526794 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e70b16-7c88-4cbb-b90f-ba8b575f987d admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.527135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e70b16-7c88-4cbb-b90f-ba8b575f987d admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.527402 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e70b16-7c88-4cbb-b90f-ba8b575f987d admin admin] PATH_INFO: `/v3/projects/6b68b886c19c45d585409651f33b8626/users/ecaf8ec105c04b919256f852da859eba/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.528558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 626/1248] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:55:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:55:44.529549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f478398-5d3b-4076-be5a-848c5f969241 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:44.539636 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f478398-5d3b-4076-be5a-848c5f969241 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:44.544278 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49afd1bf-4283-44b3-9a8c-d8fcfc704d95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:44.544887 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49afd1bf-4283-44b3-9a8c-d8fcfc704d95 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.545171 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49afd1bf-4283-44b3-9a8c-d8fcfc704d95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.545468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49afd1bf-4283-44b3-9a8c-d8fcfc704d95 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.546095 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49afd1bf-4283-44b3-9a8c-d8fcfc704d95 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:55:44.546393 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49afd1bf-4283-44b3-9a8c-d8fcfc704d95 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:44.547999 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49afd1bf-4283-44b3-9a8c-d8fcfc704d95 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:44.549863 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e70b16-7c88-4cbb-b90f-ba8b575f987d admin admin] RBAC: Authorizing `identity:create_grant(project_id=6b68b886c19c45d585409651f33b8626, user_id=ecaf8ec105c04b919256f852da859eba, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:44.552569 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e70b16-7c88-4cbb-b90f-ba8b575f987d admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:44.561458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 627/1249] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:55:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:55:44.562480 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49afd1bf-4283-44b3-9a8c-d8fcfc704d95 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:44.573644 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 623/1250] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:55:44 2022] PUT /identity/v3/projects/6b68b886c19c45d585409651f33b8626/users/ecaf8ec105c04b919256f852da859eba/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:55:44.576598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e70b16-7c88-4cbb-b90f-ba8b575f987d admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:44.577758 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49afd1bf-4283-44b3-9a8c-d8fcfc704d95 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:44.627084 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:44.634229 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939d97c6-efe9-49a4-b2bc-0133ae406380 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:44.634848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939d97c6-efe9-49a4-b2bc-0133ae406380 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.635239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939d97c6-efe9-49a4-b2bc-0133ae406380 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.635856 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939d97c6-efe9-49a4-b2bc-0133ae406380 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.636649 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939d97c6-efe9-49a4-b2bc-0133ae406380 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:44.638788 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939d97c6-efe9-49a4-b2bc-0133ae406380 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:44.639835 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e70b16-7c88-4cbb-b90f-ba8b575f987d admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:44.644311 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727ed0f2-7928-4ff2-94de-c8f2d1c6949a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:44.644922 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727ed0f2-7928-4ff2-94de-c8f2d1c6949a admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.645304 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727ed0f2-7928-4ff2-94de-c8f2d1c6949a admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.645579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727ed0f2-7928-4ff2-94de-c8f2d1c6949a admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.646202 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727ed0f2-7928-4ff2-94de-c8f2d1c6949a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:44.648681 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727ed0f2-7928-4ff2-94de-c8f2d1c6949a admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:44.666663 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 628/1251] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:55:44 2022] POST /identity/v3/projects => generated 367 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:44.667535 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939d97c6-efe9-49a4-b2bc-0133ae406380 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:44.677207 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 624/1252] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:55:44 2022] POST /identity/v3/projects => generated 385 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:44.677965 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727ed0f2-7928-4ff2-94de-c8f2d1c6949a admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:44.724353 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:44.733262 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939d97c6-efe9-49a4-b2bc-0133ae406380 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:44.743505 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727ed0f2-7928-4ff2-94de-c8f2d1c6949a admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:44.748100 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030135a6-9a6a-4de5-a583-39d2994b6923 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:44.748722 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030135a6-9a6a-4de5-a583-39d2994b6923 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.748987 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030135a6-9a6a-4de5-a583-39d2994b6923 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.749260 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030135a6-9a6a-4de5-a583-39d2994b6923 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.749882 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030135a6-9a6a-4de5-a583-39d2994b6923 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:44.751854 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030135a6-9a6a-4de5-a583-39d2994b6923 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:44.775867 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 625/1253] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:55:44 2022] POST /identity/v3/users => generated 293 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:44.775867 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030135a6-9a6a-4de5-a583-39d2994b6923 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:44.786723 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030135a6-9a6a-4de5-a583-39d2994b6923 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:44.791443 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b29a110-f729-4ec6-94bb-014e3078c4b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:44.791978 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a70347-7589-4840-aa1a-22862f9f1e20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:44.792112 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b29a110-f729-4ec6-94bb-014e3078c4b3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.792369 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b29a110-f729-4ec6-94bb-014e3078c4b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.792594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a70347-7589-4840-aa1a-22862f9f1e20 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.792724 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b29a110-f729-4ec6-94bb-014e3078c4b3 admin admin] PATH_INFO: `/v3/projects/8bf407954ed34ebfaf56db206df4fc84/users/1c0e5bd11ac14259ad376ce54bcb7025/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.792849 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a70347-7589-4840-aa1a-22862f9f1e20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.793120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a70347-7589-4840-aa1a-22862f9f1e20 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.793748 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a70347-7589-4840-aa1a-22862f9f1e20 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:44.795658 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a70347-7589-4840-aa1a-22862f9f1e20 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:44.807272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b29a110-f729-4ec6-94bb-014e3078c4b3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8bf407954ed34ebfaf56db206df4fc84, user_id=1c0e5bd11ac14259ad376ce54bcb7025, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:44.809933 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b29a110-f729-4ec6-94bb-014e3078c4b3 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:44.822963 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 629/1254] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:55:44 2022] POST /identity/v3/projects => generated 381 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:44.823957 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a70347-7589-4840-aa1a-22862f9f1e20 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:44.824649 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 626/1255] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:55:44 2022] PUT /identity/v3/projects/8bf407954ed34ebfaf56db206df4fc84/users/1c0e5bd11ac14259ad376ce54bcb7025/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:55:44.828698 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b29a110-f729-4ec6-94bb-014e3078c4b3 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:44.888126 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a70347-7589-4840-aa1a-22862f9f1e20 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:44.892266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b29a110-f729-4ec6-94bb-014e3078c4b3 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:44.892828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4bc673d-5948-4fda-b3f4-4147d81505ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:44.893493 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bc673d-5948-4fda-b3f4-4147d81505ab admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.893771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bc673d-5948-4fda-b3f4-4147d81505ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.894041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bc673d-5948-4fda-b3f4-4147d81505ab admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.894699 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bc673d-5948-4fda-b3f4-4147d81505ab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:44.896696 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bc673d-5948-4fda-b3f4-4147d81505ab admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:44.897275 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81728ea1-551c-43a1-84e5-86b618ec235c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:44.897876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81728ea1-551c-43a1-84e5-86b618ec235c admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.898145 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81728ea1-551c-43a1-84e5-86b618ec235c admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.898410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81728ea1-551c-43a1-84e5-86b618ec235c admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.899024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81728ea1-551c-43a1-84e5-86b618ec235c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:44.900927 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81728ea1-551c-43a1-84e5-86b618ec235c admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:44.919303 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:44.927357 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 630/1256] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:55:44 2022] POST /identity/v3/projects => generated 377 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:44.929355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b8c247-2467-41db-add4-aecf7d25b48d None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:44.929822 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b8c247-2467-41db-add4-aecf7d25b48d None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:44.930248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b8c247-2467-41db-add4-aecf7d25b48d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:44.933891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 627/1257] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:55:44 2022] POST /identity/v3/projects => generated 381 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:44.933891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81728ea1-551c-43a1-84e5-86b618ec235c admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:44.951678 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-48b8c247-2467-41db-add4-aecf7d25b48d None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:44.952948 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-48b8c247-2467-41db-add4-aecf7d25b48d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:44.995679 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81728ea1-551c-43a1-84e5-86b618ec235c admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.000305 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e7186f-c19f-4285-badc-9a2d19e89f4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.000917 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e7186f-c19f-4285-badc-9a2d19e89f4e admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.001190 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e7186f-c19f-4285-badc-9a2d19e89f4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.001458 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e7186f-c19f-4285-badc-9a2d19e89f4e admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.002104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e7186f-c19f-4285-badc-9a2d19e89f4e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.004093 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e7186f-c19f-4285-badc-9a2d19e89f4e admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.026123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 631/1258] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:45.026960 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b8c247-2467-41db-add4-aecf7d25b48d None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.030349 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 628/1259] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:55:44 2022] POST /identity/v3/users => generated 284 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:45.030854 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e7186f-c19f-4285-badc-9a2d19e89f4e admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.036460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b8c247-2467-41db-add4-aecf7d25b48d None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.041221 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48c546b-1861-47a4-8966-cdbe2723ed7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.041832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48c546b-1861-47a4-8966-cdbe2723ed7b admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.042159 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48c546b-1861-47a4-8966-cdbe2723ed7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.042397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48c546b-1861-47a4-8966-cdbe2723ed7b admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.042936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e7186f-c19f-4285-badc-9a2d19e89f4e admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.043073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48c546b-1861-47a4-8966-cdbe2723ed7b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.045015 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48c546b-1861-47a4-8966-cdbe2723ed7b admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.047826 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2025efec-25d5-4bbc-9357-13f7a66c8991 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.048431 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2025efec-25d5-4bbc-9357-13f7a66c8991 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.048707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2025efec-25d5-4bbc-9357-13f7a66c8991 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.048973 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2025efec-25d5-4bbc-9357-13f7a66c8991 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.049608 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2025efec-25d5-4bbc-9357-13f7a66c8991 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.051180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2025efec-25d5-4bbc-9357-13f7a66c8991 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.062344 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 629/1260] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:55:45.063136 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2025efec-25d5-4bbc-9357-13f7a66c8991 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.070303 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 632/1261] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:55:45 2022] POST /identity/v3/users => generated 293 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:45.071933 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cb93cf-ceb3-4e7a-a94d-f62c65e2eec4 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.072218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cb93cf-ceb3-4e7a-a94d-f62c65e2eec4 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.072490 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cb93cf-ceb3-4e7a-a94d-f62c65e2eec4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.073189 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2025efec-25d5-4bbc-9357-13f7a66c8991 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.077932 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636976e4-9c9a-4912-98e9-f03ff53c6aa2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.078520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636976e4-9c9a-4912-98e9-f03ff53c6aa2 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.078794 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636976e4-9c9a-4912-98e9-f03ff53c6aa2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.079065 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636976e4-9c9a-4912-98e9-f03ff53c6aa2 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.079693 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636976e4-9c9a-4912-98e9-f03ff53c6aa2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.081628 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636976e4-9c9a-4912-98e9-f03ff53c6aa2 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.091884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-56cb93cf-ceb3-4e7a-a94d-f62c65e2eec4 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:45.092080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=90,keystone:INSERT=10 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:45.093100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-56cb93cf-ceb3-4e7a-a94d-f62c65e2eec4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:45.102893 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 633/1262] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:45.103746 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cb93cf-ceb3-4e7a-a94d-f62c65e2eec4 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.110064 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 630/1263] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:55:45 2022] POST /identity/v3/users => generated 291 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:45.110064 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636976e4-9c9a-4912-98e9-f03ff53c6aa2 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.113956 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cb93cf-ceb3-4e7a-a94d-f62c65e2eec4 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.119003 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636976e4-9c9a-4912-98e9-f03ff53c6aa2 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.119870 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eeaa426-d73d-41f0-bca5-1cbed1f1921c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.120509 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eeaa426-d73d-41f0-bca5-1cbed1f1921c admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.120780 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eeaa426-d73d-41f0-bca5-1cbed1f1921c admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.121046 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eeaa426-d73d-41f0-bca5-1cbed1f1921c admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.121668 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eeaa426-d73d-41f0-bca5-1cbed1f1921c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.123584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eeaa426-d73d-41f0-bca5-1cbed1f1921c admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.123820 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f32149-046b-4acd-a7fe-32e28d57dea5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.124555 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f32149-046b-4acd-a7fe-32e28d57dea5 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.124830 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f32149-046b-4acd-a7fe-32e28d57dea5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.125096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f32149-046b-4acd-a7fe-32e28d57dea5 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.125738 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f32149-046b-4acd-a7fe-32e28d57dea5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.127676 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f32149-046b-4acd-a7fe-32e28d57dea5 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.150715 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 634/1264] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:55:45 2022] POST /identity/v3/users => generated 289 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:45.150715 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eeaa426-d73d-41f0-bca5-1cbed1f1921c admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.151040 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 631/1265] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:55:45 2022] POST /identity/v3/users => generated 291 bytes in 42 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:45.151410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f32149-046b-4acd-a7fe-32e28d57dea5 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.160835 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eeaa426-d73d-41f0-bca5-1cbed1f1921c admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.165765 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a68d79-80f0-4c48-8e49-bf1174967d36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.166408 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a68d79-80f0-4c48-8e49-bf1174967d36 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.166696 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a68d79-80f0-4c48-8e49-bf1174967d36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.166937 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a68d79-80f0-4c48-8e49-bf1174967d36 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.167564 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a68d79-80f0-4c48-8e49-bf1174967d36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.169173 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a68d79-80f0-4c48-8e49-bf1174967d36 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.179187 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 635/1266] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:55:45.180000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a68d79-80f0-4c48-8e49-bf1174967d36 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.192565 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a68d79-80f0-4c48-8e49-bf1174967d36 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.197917 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e70470-1532-4c24-8c11-47dd5fc36760 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.198517 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e70470-1532-4c24-8c11-47dd5fc36760 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.198830 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e70470-1532-4c24-8c11-47dd5fc36760 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.199092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e70470-1532-4c24-8c11-47dd5fc36760 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.199711 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e70470-1532-4c24-8c11-47dd5fc36760 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.201399 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e70470-1532-4c24-8c11-47dd5fc36760 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.206096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f32149-046b-4acd-a7fe-32e28d57dea5 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.209973 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30924cba-3a3d-4c34-b4bc-f3d6a6a2e64b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.210626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30924cba-3a3d-4c34-b4bc-f3d6a6a2e64b service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.210907 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30924cba-3a3d-4c34-b4bc-f3d6a6a2e64b service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.211176 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30924cba-3a3d-4c34-b4bc-f3d6a6a2e64b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.212314 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 636/1267] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:55:45.213117 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e70470-1532-4c24-8c11-47dd5fc36760 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.220884 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30924cba-3a3d-4c34-b4bc-f3d6a6a2e64b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.222325 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30924cba-3a3d-4c34-b4bc-f3d6a6a2e64b service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.222655 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e70470-1532-4c24-8c11-47dd5fc36760 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.227507 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b14ff1-2bab-4469-afb7-663a2445be6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.227754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b14ff1-2bab-4469-afb7-663a2445be6e admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.228027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b14ff1-2bab-4469-afb7-663a2445be6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.228296 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b14ff1-2bab-4469-afb7-663a2445be6e admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.228919 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b14ff1-2bab-4469-afb7-663a2445be6e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.230472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b14ff1-2bab-4469-afb7-663a2445be6e admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.231764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 632/1268] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:55:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:55:45.232574 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30924cba-3a3d-4c34-b4bc-f3d6a6a2e64b service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.241866 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30924cba-3a3d-4c34-b4bc-f3d6a6a2e64b service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.242115 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 637/1269] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:55:45.243132 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b14ff1-2bab-4469-afb7-663a2445be6e admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.246857 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a32134-8a1e-4476-adbd-47719a06567f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.247564 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a32134-8a1e-4476-adbd-47719a06567f admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.247882 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a32134-8a1e-4476-adbd-47719a06567f admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.248147 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a32134-8a1e-4476-adbd-47719a06567f admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.248801 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a32134-8a1e-4476-adbd-47719a06567f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.250380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a32134-8a1e-4476-adbd-47719a06567f admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.254262 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b14ff1-2bab-4469-afb7-663a2445be6e admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.259440 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8111da4f-d316-47cc-b7a5-66d87400055d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.260446 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8111da4f-d316-47cc-b7a5-66d87400055d service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.260933 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8111da4f-d316-47cc-b7a5-66d87400055d service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.261417 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8111da4f-d316-47cc-b7a5-66d87400055d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.265541 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 633/1270] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:55:45.265541 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a32134-8a1e-4476-adbd-47719a06567f admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.275024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a32134-8a1e-4476-adbd-47719a06567f admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.275328 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8111da4f-d316-47cc-b7a5-66d87400055d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.276878 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8111da4f-d316-47cc-b7a5-66d87400055d service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.279868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee71b7f-264d-4f34-988d-61697d9bd677 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.280491 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee71b7f-264d-4f34-988d-61697d9bd677 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.280767 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee71b7f-264d-4f34-988d-61697d9bd677 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.281138 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee71b7f-264d-4f34-988d-61697d9bd677 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.281661 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee71b7f-264d-4f34-988d-61697d9bd677 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.283264 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee71b7f-264d-4f34-988d-61697d9bd677 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.288840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 638/1271] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:55:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:55:45.289920 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8111da4f-d316-47cc-b7a5-66d87400055d service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.294402 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 634/1272] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:55:45.298211 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee71b7f-264d-4f34-988d-61697d9bd677 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.300719 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8111da4f-d316-47cc-b7a5-66d87400055d service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.306689 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d154bc-5512-47f5-b8cf-e70c61a8d427 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.307302 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d154bc-5512-47f5-b8cf-e70c61a8d427 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.307597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d154bc-5512-47f5-b8cf-e70c61a8d427 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.307888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d154bc-5512-47f5-b8cf-e70c61a8d427 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.308759 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d154bc-5512-47f5-b8cf-e70c61a8d427 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.310612 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d154bc-5512-47f5-b8cf-e70c61a8d427 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.311737 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee71b7f-264d-4f34-988d-61697d9bd677 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.318311 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd43fe2-cd6f-48f5-885b-cdf17077b498 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.319877 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd43fe2-cd6f-48f5-885b-cdf17077b498 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.320157 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd43fe2-cd6f-48f5-885b-cdf17077b498 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.320424 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd43fe2-cd6f-48f5-885b-cdf17077b498 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.321091 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd43fe2-cd6f-48f5-885b-cdf17077b498 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.322685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd43fe2-cd6f-48f5-885b-cdf17077b498 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.323290 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 639/1273] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:55:45.324293 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d154bc-5512-47f5-b8cf-e70c61a8d427 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.336613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d154bc-5512-47f5-b8cf-e70c61a8d427 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.337779 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 635/1274] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:55:45.342040 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd43fe2-cd6f-48f5-885b-cdf17077b498 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.347769 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4acaf98-f1ed-4c62-8412-73ddebf01a5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.348792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4acaf98-f1ed-4c62-8412-73ddebf01a5f admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.349299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4acaf98-f1ed-4c62-8412-73ddebf01a5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.350118 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4acaf98-f1ed-4c62-8412-73ddebf01a5f admin admin] PATH_INFO: `/v3/projects/a3bfc7112bb04760aa4e9adc50b12361/users/15e603f9898a430d8a53f21818e97722/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.370795 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4acaf98-f1ed-4c62-8412-73ddebf01a5f admin admin] RBAC: Authorizing `identity:create_grant(project_id=a3bfc7112bb04760aa4e9adc50b12361, user_id=15e603f9898a430d8a53f21818e97722, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.371011 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd43fe2-cd6f-48f5-885b-cdf17077b498 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.373490 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4acaf98-f1ed-4c62-8412-73ddebf01a5f admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.376271 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e531aaee-f303-4bda-b78e-6ea55bcabd01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.377213 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e531aaee-f303-4bda-b78e-6ea55bcabd01 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.377641 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e531aaee-f303-4bda-b78e-6ea55bcabd01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.377908 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e531aaee-f303-4bda-b78e-6ea55bcabd01 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.378678 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e531aaee-f303-4bda-b78e-6ea55bcabd01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.380761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e531aaee-f303-4bda-b78e-6ea55bcabd01 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.390166 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 640/1275] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:55:45 2022] PUT /identity/v3/projects/a3bfc7112bb04760aa4e9adc50b12361/users/15e603f9898a430d8a53f21818e97722/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:55:45.392845 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 636/1276] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:55:45.393840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4acaf98-f1ed-4c62-8412-73ddebf01a5f admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.393951 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e531aaee-f303-4bda-b78e-6ea55bcabd01 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.408124 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e531aaee-f303-4bda-b78e-6ea55bcabd01 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.439811 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:45.472432 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4acaf98-f1ed-4c62-8412-73ddebf01a5f admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.477355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265073c2-8b2a-4a5f-9d56-596f6ed829b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.478070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265073c2-8b2a-4a5f-9d56-596f6ed829b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.478315 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265073c2-8b2a-4a5f-9d56-596f6ed829b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.478518 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265073c2-8b2a-4a5f-9d56-596f6ed829b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.479185 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265073c2-8b2a-4a5f-9d56-596f6ed829b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.480828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265073c2-8b2a-4a5f-9d56-596f6ed829b9 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.484468 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310c7b72-fa00-4034-8ada-1015980bfa36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.485250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310c7b72-fa00-4034-8ada-1015980bfa36 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.485546 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310c7b72-fa00-4034-8ada-1015980bfa36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.485818 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310c7b72-fa00-4034-8ada-1015980bfa36 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.487148 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310c7b72-fa00-4034-8ada-1015980bfa36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.489147 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310c7b72-fa00-4034-8ada-1015980bfa36 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.491504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 641/1277] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:55:45.492179 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265073c2-8b2a-4a5f-9d56-596f6ed829b9 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.503670 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265073c2-8b2a-4a5f-9d56-596f6ed829b9 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.509407 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 637/1278] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:55:45.509407 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310c7b72-fa00-4034-8ada-1015980bfa36 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.512287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8aadb5-672d-45bd-88e0-9eb9aac2cfaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.512910 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8aadb5-672d-45bd-88e0-9eb9aac2cfaa admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.513188 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8aadb5-672d-45bd-88e0-9eb9aac2cfaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.513456 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8aadb5-672d-45bd-88e0-9eb9aac2cfaa admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.514085 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8aadb5-672d-45bd-88e0-9eb9aac2cfaa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.515672 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8aadb5-672d-45bd-88e0-9eb9aac2cfaa admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.523102 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310c7b72-fa00-4034-8ada-1015980bfa36 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.531266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e386815-9367-40ab-95b9-9697a77ad686 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.532660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 642/1279] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:55:45.534953 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307ddfa8-f714-4423-a552-206e4a3140ad None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.535426 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307ddfa8-f714-4423-a552-206e4a3140ad None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.536197 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307ddfa8-f714-4423-a552-206e4a3140ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.536573 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e386815-9367-40ab-95b9-9697a77ad686 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.537325 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e386815-9367-40ab-95b9-9697a77ad686 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.537768 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e386815-9367-40ab-95b9-9697a77ad686 admin admin] PATH_INFO: `/v3/projects/47ea8ebe1bc64f6183b8dca990bb3ae4/users/a319903641df4cb69edc2f2508b544ec/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.563082 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-307ddfa8-f714-4423-a552-206e4a3140ad None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:45.564350 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-307ddfa8-f714-4423-a552-206e4a3140ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:45.567678 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e386815-9367-40ab-95b9-9697a77ad686 admin admin] RBAC: Authorizing `identity:create_grant(project_id=47ea8ebe1bc64f6183b8dca990bb3ae4, user_id=a319903641df4cb69edc2f2508b544ec, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.570853 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e386815-9367-40ab-95b9-9697a77ad686 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.574435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 643/1280] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:45.575479 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307ddfa8-f714-4423-a552-206e4a3140ad None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.592143 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 638/1281] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:55:45 2022] PUT /identity/v3/projects/47ea8ebe1bc64f6183b8dca990bb3ae4/users/a319903641df4cb69edc2f2508b544ec/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:55:45.593025 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e386815-9367-40ab-95b9-9697a77ad686 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.605617 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:45.606114 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307ddfa8-f714-4423-a552-206e4a3140ad None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.679680 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90acee2-631c-4a0d-97d9-65cd2dff1538 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.680744 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90acee2-631c-4a0d-97d9-65cd2dff1538 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.681188 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90acee2-631c-4a0d-97d9-65cd2dff1538 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.681851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90acee2-631c-4a0d-97d9-65cd2dff1538 admin admin] PATH_INFO: `/v3/projects/ce1df4d4d27a412b91845bc0fb02c3b9/users/0038d0604f1942fdb1fcf91140d5197d/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.702679 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90acee2-631c-4a0d-97d9-65cd2dff1538 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ce1df4d4d27a412b91845bc0fb02c3b9, user_id=0038d0604f1942fdb1fcf91140d5197d, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.706183 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90acee2-631c-4a0d-97d9-65cd2dff1538 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.707249 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e386815-9367-40ab-95b9-9697a77ad686 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.721232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f44663-1ee6-4f50-b818-02ecb480fe4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.721945 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f44663-1ee6-4f50-b818-02ecb480fe4b admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.722473 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f44663-1ee6-4f50-b818-02ecb480fe4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.722881 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f44663-1ee6-4f50-b818-02ecb480fe4b admin admin] PATH_INFO: `/v3/projects/b6087ed6d73147bda7f79cf3f2b5b17f/users/e46f636ec9f44631a7f410d7702a2393/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.739175 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 644/1282] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:55:45 2022] PUT /identity/v3/projects/ce1df4d4d27a412b91845bc0fb02c3b9/users/0038d0604f1942fdb1fcf91140d5197d/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:55:45.741429 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90acee2-631c-4a0d-97d9-65cd2dff1538 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.760637 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f44663-1ee6-4f50-b818-02ecb480fe4b admin admin] RBAC: Authorizing `identity:create_grant(project_id=b6087ed6d73147bda7f79cf3f2b5b17f, user_id=e46f636ec9f44631a7f410d7702a2393, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.763696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f44663-1ee6-4f50-b818-02ecb480fe4b admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.789713 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 639/1283] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:55:45 2022] PUT /identity/v3/projects/b6087ed6d73147bda7f79cf3f2b5b17f/users/e46f636ec9f44631a7f410d7702a2393/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:55:45.800279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f44663-1ee6-4f50-b818-02ecb480fe4b admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.836629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90acee2-631c-4a0d-97d9-65cd2dff1538 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.911280 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f44663-1ee6-4f50-b818-02ecb480fe4b admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:45.920772 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f28e68-2020-4635-a5ac-93e91664b2d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.921440 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f28e68-2020-4635-a5ac-93e91664b2d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.922659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f28e68-2020-4635-a5ac-93e91664b2d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.923068 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:45.928423 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f28e68-2020-4635-a5ac-93e91664b2d5 admin admin] PATH_INFO: `/v3/projects/eb71114e972447f6bcfad53c4d2bad7e/users/874f7fbc347c4cd69a2ed9c0b18d391e/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.940781 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac2663f-006b-43b7-8f07-e8bb9f256a0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:45.941927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac2663f-006b-43b7-8f07-e8bb9f256a0d admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:45.942693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac2663f-006b-43b7-8f07-e8bb9f256a0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:45.943210 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac2663f-006b-43b7-8f07-e8bb9f256a0d admin admin] PATH_INFO: `/v3/projects/0f7d505c7b9e4ca08d6437e6e529a039/users/c92f7b4d09d74b8d93d739c1938694e6/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:45.962420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f28e68-2020-4635-a5ac-93e91664b2d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=eb71114e972447f6bcfad53c4d2bad7e, user_id=874f7fbc347c4cd69a2ed9c0b18d391e, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.966878 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac2663f-006b-43b7-8f07-e8bb9f256a0d admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f7d505c7b9e4ca08d6437e6e529a039, user_id=c92f7b4d09d74b8d93d739c1938694e6, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:45.970077 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f28e68-2020-4635-a5ac-93e91664b2d5 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.970575 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac2663f-006b-43b7-8f07-e8bb9f256a0d admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:45.989856 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 645/1284] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:55:45 2022] PUT /identity/v3/projects/0f7d505c7b9e4ca08d6437e6e529a039/users/c92f7b4d09d74b8d93d739c1938694e6/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:55:45.991917 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac2663f-006b-43b7-8f07-e8bb9f256a0d admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:45.994143 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-2ac2663f-006b-43b7-8f07-e8bb9f256a0d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:46.000356 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 640/1285] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:55:45 2022] PUT /identity/v3/projects/eb71114e972447f6bcfad53c4d2bad7e/users/874f7fbc347c4cd69a2ed9c0b18d391e/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:55:46.001715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8af45d-088a-457d-8fb6-c582d2a2eb0b None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:46.001981 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8af45d-088a-457d-8fb6-c582d2a2eb0b None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:46.004216 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8af45d-088a-457d-8fb6-c582d2a2eb0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:46.059194 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-db8af45d-088a-457d-8fb6-c582d2a2eb0b None None] MFA Rules not processed for user `a319903641df4cb69edc2f2508b544ec`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:46.060953 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-db8af45d-088a-457d-8fb6-c582d2a2eb0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:46.081410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac2663f-006b-43b7-8f07-e8bb9f256a0d admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:46.151176 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d50476d-d672-4f9a-9f6e-8c42f8688f6d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:46.151866 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d50476d-d672-4f9a-9f6e-8c42f8688f6d service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:46.152509 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d50476d-d672-4f9a-9f6e-8c42f8688f6d service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:46.152630 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d50476d-d672-4f9a-9f6e-8c42f8688f6d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:46.173196 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 641/1286] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:46 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:46.173196 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c6a3bf-42ae-4073-aa7b-971829135927 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:46.173196 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c6a3bf-42ae-4073-aa7b-971829135927 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:46.176110 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c6a3bf-42ae-4073-aa7b-971829135927 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:46.188962 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:46.227172 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-42c6a3bf-42ae-4073-aa7b-971829135927 None None] MFA Rules not processed for user `0038d0604f1942fdb1fcf91140d5197d`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:46.229647 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-42c6a3bf-42ae-4073-aa7b-971829135927 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:46.263737 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d50476d-d672-4f9a-9f6e-8c42f8688f6d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:46.266620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d50476d-d672-4f9a-9f6e-8c42f8688f6d service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:46.293339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 646/1287] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:55:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:55:46.297349 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d4dcab-1688-4efb-a78a-7e1f109c6a45 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:46.297637 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d4dcab-1688-4efb-a78a-7e1f109c6a45 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:46.297957 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d4dcab-1688-4efb-a78a-7e1f109c6a45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:46.360665 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 642/1288] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:46 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:46.363777 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e280d7b-78df-44e9-8cfb-1c23d2e505c6 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:46.365639 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e280d7b-78df-44e9-8cfb-1c23d2e505c6 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:46.366531 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e280d7b-78df-44e9-8cfb-1c23d2e505c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:46.380096 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-f1d4dcab-1688-4efb-a78a-7e1f109c6a45 None None] MFA Rules not processed for user `e46f636ec9f44631a7f410d7702a2393`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:46.381842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-f1d4dcab-1688-4efb-a78a-7e1f109c6a45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:46.411557 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-0e280d7b-78df-44e9-8cfb-1c23d2e505c6 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:46.413390 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-0e280d7b-78df-44e9-8cfb-1c23d2e505c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:46.422619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:46.433777 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 643/1289] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:46.434897 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02a9761-1349-47ed-a00b-2169f8035d45 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:46.435189 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02a9761-1349-47ed-a00b-2169f8035d45 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:46.435447 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02a9761-1349-47ed-a00b-2169f8035d45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:46.499107 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-a02a9761-1349-47ed-a00b-2169f8035d45 None None] MFA Rules not processed for user `874f7fbc347c4cd69a2ed9c0b18d391e`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:46.500458 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a02a9761-1349-47ed-a00b-2169f8035d45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:46.503211 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 647/1290] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:46 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:46.504455 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d4dcab-1688-4efb-a78a-7e1f109c6a45 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:46.645400 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 644/1291] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:46 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:46.648419 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0187416d-f9d5-4f94-9cd6-aec063fd8a64 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:46.649465 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0187416d-f9d5-4f94-9cd6-aec063fd8a64 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:46.651415 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0187416d-f9d5-4f94-9cd6-aec063fd8a64 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:46.654437 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 645/1292] 10.222.0.16 () {54 vars in 961 bytes} [Sun Oct 30 00:55:46 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:55:46.656472 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0187416d-f9d5-4f94-9cd6-aec063fd8a64 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:46.666317 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d4dcab-1688-4efb-a78a-7e1f109c6a45 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:46.676241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250169a3-9d14-402a-b3b4-17056a44bb42 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:46.677843 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250169a3-9d14-402a-b3b4-17056a44bb42 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:46.677843 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250169a3-9d14-402a-b3b4-17056a44bb42 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:46.679019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250169a3-9d14-402a-b3b4-17056a44bb42 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:46.680563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0187416d-f9d5-4f94-9cd6-aec063fd8a64 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:46.688141 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b7dd84-5d37-4fe5-8c8a-cebbaa15b086 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:46.689202 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b7dd84-5d37-4fe5-8c8a-cebbaa15b086 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:46.689688 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b7dd84-5d37-4fe5-8c8a-cebbaa15b086 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:46.690443 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b7dd84-5d37-4fe5-8c8a-cebbaa15b086 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:46.695159 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250169a3-9d14-402a-b3b4-17056a44bb42 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:46.696841 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250169a3-9d14-402a-b3b4-17056a44bb42 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:46.711983 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 648/1293] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:55:46 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:55:46.713666 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250169a3-9d14-402a-b3b4-17056a44bb42 service cinder] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:46.724439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b7dd84-5d37-4fe5-8c8a-cebbaa15b086 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:46.727285 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b7dd84-5d37-4fe5-8c8a-cebbaa15b086 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:46.728696 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250169a3-9d14-402a-b3b4-17056a44bb42 service cinder] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:46.734746 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a5a193-ffcf-4797-98bb-e72c508c9a34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:46.735430 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a5a193-ffcf-4797-98bb-e72c508c9a34 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:46.735846 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a5a193-ffcf-4797-98bb-e72c508c9a34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:46.736226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a5a193-ffcf-4797-98bb-e72c508c9a34 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:46.736936 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a5a193-ffcf-4797-98bb-e72c508c9a34 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:46.739638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a5a193-ffcf-4797-98bb-e72c508c9a34 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:46.760217 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 646/1294] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:55:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:55:46.761118 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b7dd84-5d37-4fe5-8c8a-cebbaa15b086 service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:46.783029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 649/1295] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:55:46 2022] POST /identity/v3/projects => generated 377 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:46.783029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd010799-f7f7-4a0f-b5fd-a9d435409e01 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:46.783029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd010799-f7f7-4a0f-b5fd-a9d435409e01 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:46.783029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd010799-f7f7-4a0f-b5fd-a9d435409e01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:46.786822 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b7dd84-5d37-4fe5-8c8a-cebbaa15b086 service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:46.838050 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-bd010799-f7f7-4a0f-b5fd-a9d435409e01 None None] MFA Rules not processed for user `e7a886715b5b4c1d99f0b37c58cb5417`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:46.839888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-bd010799-f7f7-4a0f-b5fd-a9d435409e01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:46.941010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b63615d-0fd8-4f03-b8c0-ed98c6b1a983 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:46.942321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b63615d-0fd8-4f03-b8c0-ed98c6b1a983 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:46.942321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b63615d-0fd8-4f03-b8c0-ed98c6b1a983 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:46.942321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b63615d-0fd8-4f03-b8c0-ed98c6b1a983 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:46.942814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b63615d-0fd8-4f03-b8c0-ed98c6b1a983 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:46.946892 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 650/1296] 10.222.0.16 () {58 vars in 1058 bytes} [Sun Oct 30 00:55:46 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:46.948124 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd010799-f7f7-4a0f-b5fd-a9d435409e01 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:46.948840 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b63615d-0fd8-4f03-b8c0-ed98c6b1a983 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:46.965936 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd010799-f7f7-4a0f-b5fd-a9d435409e01 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:46.968524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:46.982477 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee0034a-5744-405c-a841-28cfdf5a611f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:46.984930 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee0034a-5744-405c-a841-28cfdf5a611f admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:46.985312 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee0034a-5744-405c-a841-28cfdf5a611f admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:46.987152 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee0034a-5744-405c-a841-28cfdf5a611f admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:46.990915 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee0034a-5744-405c-a841-28cfdf5a611f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:46.993803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee0034a-5744-405c-a841-28cfdf5a611f admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:47.028469 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 647/1297] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:55:46 2022] POST /identity/v3/projects => generated 379 bytes in 265 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:47.028469 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b63615d-0fd8-4f03-b8c0-ed98c6b1a983 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:47.060464 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 651/1298] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:55:46 2022] POST /identity/v3/users => generated 295 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:47.060464 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee0034a-5744-405c-a841-28cfdf5a611f admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:47.080977 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee0034a-5744-405c-a841-28cfdf5a611f admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:47.093672 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:47.195504 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b63615d-0fd8-4f03-b8c0-ed98c6b1a983 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:47.211972 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9cade3-1083-4c6e-a1f1-cc79860694b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:47.212884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9cade3-1083-4c6e-a1f1-cc79860694b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:47.213288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9cade3-1083-4c6e-a1f1-cc79860694b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:47.214479 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9cade3-1083-4c6e-a1f1-cc79860694b2 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:47.216094 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9cade3-1083-4c6e-a1f1-cc79860694b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:47.216505 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065bf1bf-9d99-491c-ba6e-a779b64c1b5d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:47.217268 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065bf1bf-9d99-491c-ba6e-a779b64c1b5d service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:47.217575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065bf1bf-9d99-491c-ba6e-a779b64c1b5d service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:47.217844 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065bf1bf-9d99-491c-ba6e-a779b64c1b5d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:47.219482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9cade3-1083-4c6e-a1f1-cc79860694b2 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:47.262762 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 652/1299] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:55:47 2022] POST /identity/v3/users => generated 296 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:47.262762 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9cade3-1083-4c6e-a1f1-cc79860694b2 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:47.293046 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9cade3-1083-4c6e-a1f1-cc79860694b2 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:47.310615 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851afbe4-06d7-469c-89fb-eaa6eb772707 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:47.313942 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851afbe4-06d7-469c-89fb-eaa6eb772707 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:47.314651 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851afbe4-06d7-469c-89fb-eaa6eb772707 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:47.315869 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851afbe4-06d7-469c-89fb-eaa6eb772707 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:47.319995 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851afbe4-06d7-469c-89fb-eaa6eb772707 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:47.331278 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851afbe4-06d7-469c-89fb-eaa6eb772707 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:47.370256 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 653/1300] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:55:47 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:55:47.373019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851afbe4-06d7-469c-89fb-eaa6eb772707 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:47.379760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065bf1bf-9d99-491c-ba6e-a779b64c1b5d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:47.381160 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065bf1bf-9d99-491c-ba6e-a779b64c1b5d service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:47.403149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851afbe4-06d7-469c-89fb-eaa6eb772707 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:47.409396 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 648/1301] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:55:47 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:55:47.411547 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065bf1bf-9d99-491c-ba6e-a779b64c1b5d service nova] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:47.415490 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176281be-6724-4cab-91e1-8fb1c0709cfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:47.416863 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176281be-6724-4cab-91e1-8fb1c0709cfc admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:47.417136 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176281be-6724-4cab-91e1-8fb1c0709cfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:47.419094 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176281be-6724-4cab-91e1-8fb1c0709cfc admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:47.419863 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176281be-6724-4cab-91e1-8fb1c0709cfc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:47.422991 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176281be-6724-4cab-91e1-8fb1c0709cfc admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:47.442340 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065bf1bf-9d99-491c-ba6e-a779b64c1b5d service nova] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:47.456672 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2eacfca-02f3-4d2a-8155-fe8a83c97b27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:47.457439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2eacfca-02f3-4d2a-8155-fe8a83c97b27 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:47.460274 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2eacfca-02f3-4d2a-8155-fe8a83c97b27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:47.460274 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2eacfca-02f3-4d2a-8155-fe8a83c97b27 admin admin] PATH_INFO: `/v3/projects/67b126a8520340a9a61619733d1c1ec7/users/8dddad8b7fa74e948ae27012bf7e5910/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:47.483269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 654/1302] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:55:47 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:55:47.485721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176281be-6724-4cab-91e1-8fb1c0709cfc admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:47.503510 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2eacfca-02f3-4d2a-8155-fe8a83c97b27 admin admin] RBAC: Authorizing `identity:create_grant(project_id=67b126a8520340a9a61619733d1c1ec7, user_id=8dddad8b7fa74e948ae27012bf7e5910, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:47.506679 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2eacfca-02f3-4d2a-8155-fe8a83c97b27 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:47.509243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176281be-6724-4cab-91e1-8fb1c0709cfc admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:47.523648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe97080-fc4b-42d6-b9e8-543d01e4498c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:47.524494 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe97080-fc4b-42d6-b9e8-543d01e4498c admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:47.524864 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe97080-fc4b-42d6-b9e8-543d01e4498c admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:47.526633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe97080-fc4b-42d6-b9e8-543d01e4498c admin admin] PATH_INFO: `/v3/projects/00101170482f49fe9093143e4d5a4d05/users/ffbdb986ac7f48168b83888558fc23d7/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:47.533913 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 649/1303] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:55:47 2022] PUT /identity/v3/projects/67b126a8520340a9a61619733d1c1ec7/users/8dddad8b7fa74e948ae27012bf7e5910/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:55:47.539392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2eacfca-02f3-4d2a-8155-fe8a83c97b27 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:47.572522 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe97080-fc4b-42d6-b9e8-543d01e4498c admin admin] RBAC: Authorizing `identity:create_grant(project_id=00101170482f49fe9093143e4d5a4d05, user_id=ffbdb986ac7f48168b83888558fc23d7, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:47.583270 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe97080-fc4b-42d6-b9e8-543d01e4498c admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:47.615802 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 655/1304] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:55:47 2022] PUT /identity/v3/projects/00101170482f49fe9093143e4d5a4d05/users/ffbdb986ac7f48168b83888558fc23d7/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:55:47.617367 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe97080-fc4b-42d6-b9e8-543d01e4498c admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:47.731069 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2eacfca-02f3-4d2a-8155-fe8a83c97b27 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:47.743539 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:47.800388 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe97080-fc4b-42d6-b9e8-543d01e4498c admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:47.816561 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2ea433-ff2f-4b9e-9f2c-b8534ab0cda6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:47.820096 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2ea433-ff2f-4b9e-9f2c-b8534ab0cda6 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:47.820389 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2ea433-ff2f-4b9e-9f2c-b8534ab0cda6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:47.821260 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2ea433-ff2f-4b9e-9f2c-b8534ab0cda6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:47.823238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2ea433-ff2f-4b9e-9f2c-b8534ab0cda6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:47.825518 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2ea433-ff2f-4b9e-9f2c-b8534ab0cda6 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:47.863821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 656/1305] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:55:47 2022] GET /identity/v3/roles => generated 1420 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:55:47.863821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45c068f-3063-49e7-bdef-b22653cf1902 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:47.865621 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45c068f-3063-49e7-bdef-b22653cf1902 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:47.865621 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45c068f-3063-49e7-bdef-b22653cf1902 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:47.870794 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 657/1306] 10.222.0.16 () {54 vars in 912 bytes} [Sun Oct 30 00:55:47 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:55:47.872288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45c068f-3063-49e7-bdef-b22653cf1902 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:47.883746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1f2f98-d2a9-4aba-b616-8ffb855a3496 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:47.885831 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1f2f98-d2a9-4aba-b616-8ffb855a3496 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:47.886540 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1f2f98-d2a9-4aba-b616-8ffb855a3496 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:47.887134 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1f2f98-d2a9-4aba-b616-8ffb855a3496 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:47.888217 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1f2f98-d2a9-4aba-b616-8ffb855a3496 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:47.890362 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1f2f98-d2a9-4aba-b616-8ffb855a3496 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:47.904397 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 650/1307] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:55:47 2022] GET /identity/v3/roles => generated 1420 bytes in 367 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:55:47.906457 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8f3df6-cd0f-4472-b20a-826ca256e621 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:47.906983 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8f3df6-cd0f-4472-b20a-826ca256e621 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:47.907436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8f3df6-cd0f-4472-b20a-826ca256e621 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:47.913421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45c068f-3063-49e7-bdef-b22653cf1902 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:47.924959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a30fce-d2d6-439e-8385-c3fd58c40cf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:47.925638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a30fce-d2d6-439e-8385-c3fd58c40cf6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:47.925925 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a30fce-d2d6-439e-8385-c3fd58c40cf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:47.926335 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a30fce-d2d6-439e-8385-c3fd58c40cf6 admin admin] PATH_INFO: `/v3/projects/00101170482f49fe9093143e4d5a4d05/users/ffbdb986ac7f48168b83888558fc23d7/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:47.932191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a30fce-d2d6-439e-8385-c3fd58c40cf6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=00101170482f49fe9093143e4d5a4d05, user_id=ffbdb986ac7f48168b83888558fc23d7, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:47.933101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-1c8f3df6-cd0f-4472-b20a-826ca256e621 None None] MFA Rules not processed for user `e7a886715b5b4c1d99f0b37c58cb5417`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:47.934745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-1c8f3df6-cd0f-4472-b20a-826ca256e621 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:47.935009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a30fce-d2d6-439e-8385-c3fd58c40cf6 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:47.971058 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 658/1308] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:55:47 2022] PUT /identity/v3/projects/00101170482f49fe9093143e4d5a4d05/users/ffbdb986ac7f48168b83888558fc23d7/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:55:47.971984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a30fce-d2d6-439e-8385-c3fd58c40cf6 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:48.059228 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:48.072593 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 651/1309] 10.222.0.16 () {58 vars in 1009 bytes} [Sun Oct 30 00:55:47 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:48.097549 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c643bfd8-a046-469a-a205-bb6f06023f57 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:48.097878 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c643bfd8-a046-469a-a205-bb6f06023f57 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:48.098158 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c643bfd8-a046-469a-a205-bb6f06023f57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:48.117451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a30fce-d2d6-439e-8385-c3fd58c40cf6 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:48.123456 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a36417f-739c-41af-b02c-f22b33f45415 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:48.124180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a36417f-739c-41af-b02c-f22b33f45415 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:48.124456 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a36417f-739c-41af-b02c-f22b33f45415 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:48.125151 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a36417f-739c-41af-b02c-f22b33f45415 admin admin] PATH_INFO: `/v3/projects/67b126a8520340a9a61619733d1c1ec7/users/8dddad8b7fa74e948ae27012bf7e5910/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:48.127392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a36417f-739c-41af-b02c-f22b33f45415 admin admin] RBAC: Authorizing `identity:create_grant(project_id=67b126a8520340a9a61619733d1c1ec7, user_id=8dddad8b7fa74e948ae27012bf7e5910, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:48.131402 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a36417f-739c-41af-b02c-f22b33f45415 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:48.146217 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 659/1310] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:55:47 2022] PUT /identity/v3/projects/67b126a8520340a9a61619733d1c1ec7/users/8dddad8b7fa74e948ae27012bf7e5910/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:55:48.147343 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a36417f-739c-41af-b02c-f22b33f45415 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:48.165453 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-c643bfd8-a046-469a-a205-bb6f06023f57 None None] MFA Rules not processed for user `ffbdb986ac7f48168b83888558fc23d7`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:48.173903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-c643bfd8-a046-469a-a205-bb6f06023f57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:48.230326 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a36417f-739c-41af-b02c-f22b33f45415 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:48.234431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a92111-476e-4e41-8c6b-7491628b28f4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:48.235114 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a92111-476e-4e41-8c6b-7491628b28f4 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:48.235386 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a92111-476e-4e41-8c6b-7491628b28f4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:48.235653 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a92111-476e-4e41-8c6b-7491628b28f4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:48.329140 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 652/1311] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:48 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 255 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:48.330883 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222a8563-439d-4249-adff-16c6a14adc45 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:48.331613 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222a8563-439d-4249-adff-16c6a14adc45 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:48.332132 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222a8563-439d-4249-adff-16c6a14adc45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:48.379122 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a92111-476e-4e41-8c6b-7491628b28f4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:48.382932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a92111-476e-4e41-8c6b-7491628b28f4 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:48.401674 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-222a8563-439d-4249-adff-16c6a14adc45 None None] MFA Rules not processed for user `8dddad8b7fa74e948ae27012bf7e5910`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:48.403005 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-222a8563-439d-4249-adff-16c6a14adc45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:48.404950 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 660/1312] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:55:48 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 258 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:55:48.405814 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a92111-476e-4e41-8c6b-7491628b28f4 service cinder] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:48.415098 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a92111-476e-4e41-8c6b-7491628b28f4 service cinder] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:48.419326 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c49e41-5166-4600-89b1-7adc1a321b90 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:48.420628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c49e41-5166-4600-89b1-7adc1a321b90 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:48.421055 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c49e41-5166-4600-89b1-7adc1a321b90 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:48.421665 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c49e41-5166-4600-89b1-7adc1a321b90 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:48.431462 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:48.433906 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c49e41-5166-4600-89b1-7adc1a321b90 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:48.436271 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c49e41-5166-4600-89b1-7adc1a321b90 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:48.454752 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 661/1313] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:55:48 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:55:48.597176 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 653/1314] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:48 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:48.709059 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c49e41-5166-4600-89b1-7adc1a321b90 service cinder] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:48.719216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c49e41-5166-4600-89b1-7adc1a321b90 service cinder] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:48.734499 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45c0768-73d4-492d-8456-2b6a78af2433 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:48.735373 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45c0768-73d4-492d-8456-2b6a78af2433 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:48.736034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45c0768-73d4-492d-8456-2b6a78af2433 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:48.736710 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45c0768-73d4-492d-8456-2b6a78af2433 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:48.770088 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45c0768-73d4-492d-8456-2b6a78af2433 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:48.772955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45c0768-73d4-492d-8456-2b6a78af2433 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:48.796196 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 662/1315] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:55:48 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:55:48.859104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222a8563-439d-4249-adff-16c6a14adc45 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:48.872404 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222a8563-439d-4249-adff-16c6a14adc45 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:48.884836 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c506e7-ea2e-400c-9c10-be95a50421cd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:48.888193 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c506e7-ea2e-400c-9c10-be95a50421cd service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:48.888506 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c506e7-ea2e-400c-9c10-be95a50421cd service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:48.889243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c506e7-ea2e-400c-9c10-be95a50421cd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:49.040687 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c506e7-ea2e-400c-9c10-be95a50421cd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:49.042487 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c506e7-ea2e-400c-9c10-be95a50421cd service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:49.062595 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 654/1316] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:55:48 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:55:49.818667 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f3c447-eba4-4bac-85c9-194085be3fb3 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:49.819566 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f3c447-eba4-4bac-85c9-194085be3fb3 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:49.819566 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f3c447-eba4-4bac-85c9-194085be3fb3 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:49.821194 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 663/1317] 10.222.0.16 () {54 vars in 961 bytes} [Sun Oct 30 00:55:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:55:49.849817 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9c1a35-e1b4-49ad-9cf1-1d25aa9b3329 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:49.850420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9c1a35-e1b4-49ad-9cf1-1d25aa9b3329 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:49.851012 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9c1a35-e1b4-49ad-9cf1-1d25aa9b3329 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:49.899698 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-1a9c1a35-e1b4-49ad-9cf1-1d25aa9b3329 None None] MFA Rules not processed for user `e7a886715b5b4c1d99f0b37c58cb5417`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:49.904858 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-1a9c1a35-e1b4-49ad-9cf1-1d25aa9b3329 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:49.929390 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 655/1318] 10.222.0.16 () {58 vars in 1058 bytes} [Sun Oct 30 00:55:49 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:49.937523 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc1bb96-cd3f-4d66-b325-43c073a83824 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:49.938283 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc1bb96-cd3f-4d66-b325-43c073a83824 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:49.938709 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc1bb96-cd3f-4d66-b325-43c073a83824 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:49.940288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 664/1319] 10.222.0.16 () {54 vars in 912 bytes} [Sun Oct 30 00:55:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:55:49.947588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbbbe4f-510f-4c75-a72d-d486ef935333 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:49.948116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbbbe4f-510f-4c75-a72d-d486ef935333 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:49.948438 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbbbe4f-510f-4c75-a72d-d486ef935333 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:50.016524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-8fbbbe4f-510f-4c75-a72d-d486ef935333 None None] MFA Rules not processed for user `e7a886715b5b4c1d99f0b37c58cb5417`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:50.018859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-8fbbbe4f-510f-4c75-a72d-d486ef935333 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:50.046315 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc1bb96-cd3f-4d66-b325-43c073a83824 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:50.064748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 656/1320] 10.222.0.16 () {58 vars in 1009 bytes} [Sun Oct 30 00:55:49 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:50.072665 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc1bb96-cd3f-4d66-b325-43c073a83824 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:50.088800 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c0a4c2-4bed-4d40-a4e9-65acff8b7657 tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dddad8b7fa74e948ae27012bf7e5910', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '67b126a8520340a9a61619733d1c1ec7', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:50.089472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c0a4c2-4bed-4d40-a4e9-65acff8b7657 tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:50.089762 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c0a4c2-4bed-4d40-a4e9-65acff8b7657 tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:50.090172 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c0a4c2-4bed-4d40-a4e9-65acff8b7657 tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:50.090829 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c0a4c2-4bed-4d40-a4e9-65acff8b7657 tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:55:50.091138 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c0a4c2-4bed-4d40-a4e9-65acff8b7657 tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:50.103653 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c0a4c2-4bed-4d40-a4e9-65acff8b7657 tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:50.126260 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 665/1321] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:55:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:55:50.126780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbbbe4f-510f-4c75-a72d-d486ef935333 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:50.154925 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c0a4c2-4bed-4d40-a4e9-65acff8b7657 tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:50.180344 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c0a4c2-4bed-4d40-a4e9-65acff8b7657 tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:50.181007 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbbbe4f-510f-4c75-a72d-d486ef935333 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:50.199728 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470054a1-4195-4eda-89cf-0c3ba3793a84 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:50.201401 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8d5c9c-c66b-4c5b-82ce-c4744bf45ecc tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dddad8b7fa74e948ae27012bf7e5910', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '67b126a8520340a9a61619733d1c1ec7', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:50.201672 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470054a1-4195-4eda-89cf-0c3ba3793a84 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:50.202062 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8d5c9c-c66b-4c5b-82ce-c4744bf45ecc tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:50.202344 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8d5c9c-c66b-4c5b-82ce-c4744bf45ecc tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:50.202577 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470054a1-4195-4eda-89cf-0c3ba3793a84 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:50.202685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8d5c9c-c66b-4c5b-82ce-c4744bf45ecc tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:50.205168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470054a1-4195-4eda-89cf-0c3ba3793a84 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:50.206026 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8d5c9c-c66b-4c5b-82ce-c4744bf45ecc tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:50.216385 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8d5c9c-c66b-4c5b-82ce-c4744bf45ecc tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:50.249459 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470054a1-4195-4eda-89cf-0c3ba3793a84 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:50.253283 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470054a1-4195-4eda-89cf-0c3ba3793a84 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:50.279516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 657/1322] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:55:50 2022] POST /identity/v3/projects => generated 353 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:50.284308 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8d5c9c-c66b-4c5b-82ce-c4744bf45ecc tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:50.314050 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 666/1323] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:55:50 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:55:50.475406 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8d5c9c-c66b-4c5b-82ce-c4744bf45ecc tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:50.488706 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841496b3-f97d-4e5a-a57e-c302f42e03d7 tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dddad8b7fa74e948ae27012bf7e5910', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '67b126a8520340a9a61619733d1c1ec7', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:50.489341 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841496b3-f97d-4e5a-a57e-c302f42e03d7 tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:50.489621 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841496b3-f97d-4e5a-a57e-c302f42e03d7 tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:50.489893 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841496b3-f97d-4e5a-a57e-c302f42e03d7 tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:50.491031 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841496b3-f97d-4e5a-a57e-c302f42e03d7 tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:55:50.491031 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841496b3-f97d-4e5a-a57e-c302f42e03d7 tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:50.492721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841496b3-f97d-4e5a-a57e-c302f42e03d7 tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:50.530079 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 658/1324] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:55:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:55:50.746017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470054a1-4195-4eda-89cf-0c3ba3793a84 service cinder] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:50.757917 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470054a1-4195-4eda-89cf-0c3ba3793a84 service cinder] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:50.770411 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7987e223-feeb-4083-9ee0-0b4f467c9f0a tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8dddad8b7fa74e948ae27012bf7e5910', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '67b126a8520340a9a61619733d1c1ec7', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:50.772086 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7987e223-feeb-4083-9ee0-0b4f467c9f0a tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:50.772576 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7987e223-feeb-4083-9ee0-0b4f467c9f0a tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:50.773183 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7987e223-feeb-4083-9ee0-0b4f467c9f0a tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] PATH_INFO: `/v3/projects/4aec7cf4a8b64048a561e351155dd436` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:50.774677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7987e223-feeb-4083-9ee0-0b4f467c9f0a tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] RBAC: Authorizing `identity:delete_project(project_id=4aec7cf4a8b64048a561e351155dd436)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:50.778020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7987e223-feeb-4083-9ee0-0b4f467c9f0a tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:50.805216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:50.892968 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7987e223-feeb-4083-9ee0-0b4f467c9f0a tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4aec7cf4a8b64048a561e351155dd436', 'request_id': 'req-7987e223-feeb-4083-9ee0-0b4f467c9f0a'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:50.914939 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7987e223-feeb-4083-9ee0-0b4f467c9f0a tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4aec7cf4a8b64048a561e351155dd436', 'request_id': 'req-7987e223-feeb-4083-9ee0-0b4f467c9f0a'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:50.929281 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7987e223-feeb-4083-9ee0-0b4f467c9f0a tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4aec7cf4a8b64048a561e351155dd436', 'request_id': 'req-7987e223-feeb-4083-9ee0-0b4f467c9f0a'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:50.982766 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7987e223-feeb-4083-9ee0-0b4f467c9f0a tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] The token cache is being invalidate because project 4aec7cf4a8b64048a561e351155dd436 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:55:50.983059 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7987e223-feeb-4083-9ee0-0b4f467c9f0a tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:50.983831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7987e223-feeb-4083-9ee0-0b4f467c9f0a tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:50.989451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 667/1325] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:55:50 2022] DELETE /identity/v3/projects/4aec7cf4a8b64048a561e351155dd436 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:55:51.718964 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841496b3-f97d-4e5a-a57e-c302f42e03d7 tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:51.721886 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-841496b3-f97d-4e5a-a57e-c302f42e03d7 tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:51.729995 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:51.813575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841496b3-f97d-4e5a-a57e-c302f42e03d7 tempest-VolumeQuotaClassesTest-93917145 tempest-VolumeQuotaClassesTest-93917145-project-admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:51.882662 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84dc9b4f-b1b3-4b4f-bd0c-5d0dfdfd185f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:51.883620 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84dc9b4f-b1b3-4b4f-bd0c-5d0dfdfd185f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:51.884022 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84dc9b4f-b1b3-4b4f-bd0c-5d0dfdfd185f admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:51.884363 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84dc9b4f-b1b3-4b4f-bd0c-5d0dfdfd185f admin admin] PATH_INFO: `/v3/users/e46f636ec9f44631a7f410d7702a2393` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:51.885621 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84dc9b4f-b1b3-4b4f-bd0c-5d0dfdfd185f admin admin] RBAC: Authorizing `identity:delete_user(user_id=e46f636ec9f44631a7f410d7702a2393)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:51.888343 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84dc9b4f-b1b3-4b4f-bd0c-5d0dfdfd185f admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:52.008922 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-84dc9b4f-b1b3-4b4f-bd0c-5d0dfdfd185f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e46f636ec9f44631a7f410d7702a2393'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:52.025907 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-84dc9b4f-b1b3-4b4f-bd0c-5d0dfdfd185f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e46f636ec9f44631a7f410d7702a2393'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:52.026706 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-84dc9b4f-b1b3-4b4f-bd0c-5d0dfdfd185f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e46f636ec9f44631a7f410d7702a2393'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:52.043202 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-84dc9b4f-b1b3-4b4f-bd0c-5d0dfdfd185f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e46f636ec9f44631a7f410d7702a2393'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:52.064022 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-84dc9b4f-b1b3-4b4f-bd0c-5d0dfdfd185f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e46f636ec9f44631a7f410d7702a2393'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:52.078922 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 659/1326] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:55:51 2022] DELETE /identity/v3/users/e46f636ec9f44631a7f410d7702a2393 => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:55:52.083304 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d19c81f-d66d-49ee-a977-1f16134e5a17 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:52.084389 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d19c81f-d66d-49ee-a977-1f16134e5a17 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:52.085071 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d19c81f-d66d-49ee-a977-1f16134e5a17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:52.124799 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-6d19c81f-d66d-49ee-a977-1f16134e5a17 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:52.128147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-6d19c81f-d66d-49ee-a977-1f16134e5a17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:52.324354 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 668/1327] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:52.356164 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84dc9b4f-b1b3-4b4f-bd0c-5d0dfdfd185f admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:52.357622 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-84dc9b4f-b1b3-4b4f-bd0c-5d0dfdfd185f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:52.447873 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:52.511340 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84dc9b4f-b1b3-4b4f-bd0c-5d0dfdfd185f admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:52.524718 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc05e4e0-882e-4b76-803d-31850e8de2f9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:52.525739 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc05e4e0-882e-4b76-803d-31850e8de2f9 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:52.526221 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc05e4e0-882e-4b76-803d-31850e8de2f9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:52.526695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc05e4e0-882e-4b76-803d-31850e8de2f9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:52.573127 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc05e4e0-882e-4b76-803d-31850e8de2f9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:52.574906 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc05e4e0-882e-4b76-803d-31850e8de2f9 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:52.604828 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 660/1328] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:55:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 249 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:55:54.621277 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d19c81f-d66d-49ee-a977-1f16134e5a17 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:54.625717 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-6d19c81f-d66d-49ee-a977-1f16134e5a17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:54.656053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d19c81f-d66d-49ee-a977-1f16134e5a17 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:54.672491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6cc862-b7c3-40c7-bfa4-f5a9399651f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:54.673168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6cc862-b7c3-40c7-bfa4-f5a9399651f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:54.673446 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6cc862-b7c3-40c7-bfa4-f5a9399651f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:54.686296 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6cc862-b7c3-40c7-bfa4-f5a9399651f2 admin admin] PATH_INFO: `/v3/projects/b6087ed6d73147bda7f79cf3f2b5b17f` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:54.702404 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6cc862-b7c3-40c7-bfa4-f5a9399651f2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b6087ed6d73147bda7f79cf3f2b5b17f)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:54.704518 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6cc862-b7c3-40c7-bfa4-f5a9399651f2 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:54.928851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7b6cc862-b7c3-40c7-bfa4-f5a9399651f2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b6087ed6d73147bda7f79cf3f2b5b17f', 'request_id': 'req-7b6cc862-b7c3-40c7-bfa4-f5a9399651f2'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:54.955649 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7b6cc862-b7c3-40c7-bfa4-f5a9399651f2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b6087ed6d73147bda7f79cf3f2b5b17f', 'request_id': 'req-7b6cc862-b7c3-40c7-bfa4-f5a9399651f2'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:54.986368 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7b6cc862-b7c3-40c7-bfa4-f5a9399651f2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b6087ed6d73147bda7f79cf3f2b5b17f', 'request_id': 'req-7b6cc862-b7c3-40c7-bfa4-f5a9399651f2'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:55.045389 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7b6cc862-b7c3-40c7-bfa4-f5a9399651f2 admin admin] The token cache is being invalidate because project b6087ed6d73147bda7f79cf3f2b5b17f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:55:55.047677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7b6cc862-b7c3-40c7-bfa4-f5a9399651f2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:55.049515 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7b6cc862-b7c3-40c7-bfa4-f5a9399651f2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:55.055651 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 669/1329] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:55:54 2022] DELETE /identity/v3/projects/b6087ed6d73147bda7f79cf3f2b5b17f => generated 0 bytes in 435 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:55:55.059711 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc05e4e0-882e-4b76-803d-31850e8de2f9 service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:55.074170 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc05e4e0-882e-4b76-803d-31850e8de2f9 service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:55.245970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2cbdfb-ec8d-4128-845c-3da7a23ce283 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:55.247497 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2cbdfb-ec8d-4128-845c-3da7a23ce283 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:55.249624 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2cbdfb-ec8d-4128-845c-3da7a23ce283 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:55.249624 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2cbdfb-ec8d-4128-845c-3da7a23ce283 admin admin] PATH_INFO: `/v3/users/8dddad8b7fa74e948ae27012bf7e5910` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:55.252123 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2cbdfb-ec8d-4128-845c-3da7a23ce283 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8dddad8b7fa74e948ae27012bf7e5910)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:55.256914 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2cbdfb-ec8d-4128-845c-3da7a23ce283 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:55.341584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-1e2cbdfb-ec8d-4128-845c-3da7a23ce283 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8dddad8b7fa74e948ae27012bf7e5910'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:55.354648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-1e2cbdfb-ec8d-4128-845c-3da7a23ce283 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8dddad8b7fa74e948ae27012bf7e5910'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:55.355783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-1e2cbdfb-ec8d-4128-845c-3da7a23ce283 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8dddad8b7fa74e948ae27012bf7e5910'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:55.381502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-1e2cbdfb-ec8d-4128-845c-3da7a23ce283 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8dddad8b7fa74e948ae27012bf7e5910'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:55.400125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-1e2cbdfb-ec8d-4128-845c-3da7a23ce283 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8dddad8b7fa74e948ae27012bf7e5910'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:55.408064 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 661/1330] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:55:55 2022] DELETE /identity/v3/users/8dddad8b7fa74e948ae27012bf7e5910 => generated 0 bytes in 349 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:55:58.321149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6cc862-b7c3-40c7-bfa4-f5a9399651f2 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:58.322698 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-7b6cc862-b7c3-40c7-bfa4-f5a9399651f2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:58.350306 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6cc862-b7c3-40c7-bfa4-f5a9399651f2 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:58.401587 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:58.541427 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5f0008-d7fc-4eec-b3d1-15d9e03ce45e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:58.543146 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5f0008-d7fc-4eec-b3d1-15d9e03ce45e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:58.545658 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5f0008-d7fc-4eec-b3d1-15d9e03ce45e admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:58.545946 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5f0008-d7fc-4eec-b3d1-15d9e03ce45e admin admin] PATH_INFO: `/v3/projects/67b126a8520340a9a61619733d1c1ec7` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:58.550163 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5f0008-d7fc-4eec-b3d1-15d9e03ce45e admin admin] RBAC: Authorizing `identity:delete_project(project_id=67b126a8520340a9a61619733d1c1ec7)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:58.579308 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5f0008-d7fc-4eec-b3d1-15d9e03ce45e admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:58.620446 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a419baca-304f-4215-b8a7-0dd48ffbf88e None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:58.620958 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a419baca-304f-4215-b8a7-0dd48ffbf88e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:58.621175 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a419baca-304f-4215-b8a7-0dd48ffbf88e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:58.701664 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-a419baca-304f-4215-b8a7-0dd48ffbf88e None None] MFA Rules not processed for user `ecaf8ec105c04b919256f852da859eba`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:58.703066 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a419baca-304f-4215-b8a7-0dd48ffbf88e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:58.743369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ec5f0008-d7fc-4eec-b3d1-15d9e03ce45e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '67b126a8520340a9a61619733d1c1ec7', 'request_id': 'req-ec5f0008-d7fc-4eec-b3d1-15d9e03ce45e'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:58.766394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ec5f0008-d7fc-4eec-b3d1-15d9e03ce45e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '67b126a8520340a9a61619733d1c1ec7', 'request_id': 'req-ec5f0008-d7fc-4eec-b3d1-15d9e03ce45e'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:58.783933 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ec5f0008-d7fc-4eec-b3d1-15d9e03ce45e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '67b126a8520340a9a61619733d1c1ec7', 'request_id': 'req-ec5f0008-d7fc-4eec-b3d1-15d9e03ce45e'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:58.820154 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ec5f0008-d7fc-4eec-b3d1-15d9e03ce45e admin admin] The token cache is being invalidate because project 67b126a8520340a9a61619733d1c1ec7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:55:58.820443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ec5f0008-d7fc-4eec-b3d1-15d9e03ce45e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:58.821169 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ec5f0008-d7fc-4eec-b3d1-15d9e03ce45e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:55:58.823006 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 670/1331] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:55:58 2022] DELETE /identity/v3/projects/67b126a8520340a9a61619733d1c1ec7 => generated 0 bytes in 502 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:55:58.828248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89f35e3-c53f-4b01-977f-2f15cbf1db62 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:58.828470 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89f35e3-c53f-4b01-977f-2f15cbf1db62 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:58.828743 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89f35e3-c53f-4b01-977f-2f15cbf1db62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:58.858274 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:58.873664 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-b89f35e3-c53f-4b01-977f-2f15cbf1db62 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:58.874958 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-b89f35e3-c53f-4b01-977f-2f15cbf1db62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:58.946948 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 662/1332] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:58 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 319 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:55:58.962228 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a419baca-304f-4215-b8a7-0dd48ffbf88e None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:58.963598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a419baca-304f-4215-b8a7-0dd48ffbf88e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:59.064394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 671/1333] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:55:59.065981 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3eddeca-d884-4837-bf82-c367d4ae0c79 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:59.066269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3eddeca-d884-4837-bf82-c367d4ae0c79 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:59.066543 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3eddeca-d884-4837-bf82-c367d4ae0c79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:59.099166 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-f3eddeca-d884-4837-bf82-c367d4ae0c79 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:59.100963 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-f3eddeca-d884-4837-bf82-c367d4ae0c79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:59.123277 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 672/1334] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:59.124399 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3eddeca-d884-4837-bf82-c367d4ae0c79 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:59.135528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3eddeca-d884-4837-bf82-c367d4ae0c79 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:59.142012 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c7698e-d36c-43e8-948d-8d9fb3d3f76f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:59.142416 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a419baca-304f-4215-b8a7-0dd48ffbf88e None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:59.142728 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c7698e-d36c-43e8-948d-8d9fb3d3f76f admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:59.143074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c7698e-d36c-43e8-948d-8d9fb3d3f76f admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:59.143461 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c7698e-d36c-43e8-948d-8d9fb3d3f76f admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:59.144216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c7698e-d36c-43e8-948d-8d9fb3d3f76f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:55:59.144579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c7698e-d36c-43e8-948d-8d9fb3d3f76f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:59.147803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c7698e-d36c-43e8-948d-8d9fb3d3f76f admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:59.168746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2349c0e2-2443-43d4-84bd-c741cc41cd1b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:59.169417 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2349c0e2-2443-43d4-84bd-c741cc41cd1b service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:59.169693 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2349c0e2-2443-43d4-84bd-c741cc41cd1b service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:59.169961 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2349c0e2-2443-43d4-84bd-c741cc41cd1b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:59.175758 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 673/1335] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:55:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:55:59.179413 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c7698e-d36c-43e8-948d-8d9fb3d3f76f admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:59.186244 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2349c0e2-2443-43d4-84bd-c741cc41cd1b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:59.188394 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2349c0e2-2443-43d4-84bd-c741cc41cd1b service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:59.207814 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c7698e-d36c-43e8-948d-8d9fb3d3f76f admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:59.215130 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbc64c2-76b3-445e-9ca4-77374e329747 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:59.215858 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbc64c2-76b3-445e-9ca4-77374e329747 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:59.216278 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbc64c2-76b3-445e-9ca4-77374e329747 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:59.216549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbc64c2-76b3-445e-9ca4-77374e329747 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:59.217327 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bbc64c2-76b3-445e-9ca4-77374e329747 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:59.220497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bbc64c2-76b3-445e-9ca4-77374e329747 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:59.222273 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 663/1336] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:55:58 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 265 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:55:59.281618 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 674/1337] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:55:59 2022] POST /identity/v3/projects => generated 381 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:59.282945 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2349c0e2-2443-43d4-84bd-c741cc41cd1b service nova] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:59.307091 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2349c0e2-2443-43d4-84bd-c741cc41cd1b service nova] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:59.473699 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99bf0588-b872-45e1-b12b-20f2d643eadf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:59.474352 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bf0588-b872-45e1-b12b-20f2d643eadf admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:59.474922 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bf0588-b872-45e1-b12b-20f2d643eadf admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:59.475305 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bf0588-b872-45e1-b12b-20f2d643eadf admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:59.476298 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99bf0588-b872-45e1-b12b-20f2d643eadf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:59.478308 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99bf0588-b872-45e1-b12b-20f2d643eadf admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:59.517816 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 664/1338] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:55:59 2022] POST /identity/v3/users => generated 291 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:55:59.522685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbc64c2-76b3-445e-9ca4-77374e329747 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:59.534770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbc64c2-76b3-445e-9ca4-77374e329747 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:59.542796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15062ad-0e7f-421d-a8d1-6daf4f5d5f99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:59.551149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15062ad-0e7f-421d-a8d1-6daf4f5d5f99 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:59.552660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15062ad-0e7f-421d-a8d1-6daf4f5d5f99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:59.554357 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15062ad-0e7f-421d-a8d1-6daf4f5d5f99 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:59.560086 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15062ad-0e7f-421d-a8d1-6daf4f5d5f99 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:59.573634 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15062ad-0e7f-421d-a8d1-6daf4f5d5f99 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:59.583658 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:59.599826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 675/1339] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:55:59 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:55:59.603665 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4980315-614c-49f8-86ed-e3e72d0c70f3 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:59.604032 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4980315-614c-49f8-86ed-e3e72d0c70f3 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:59.604462 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4980315-614c-49f8-86ed-e3e72d0c70f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:59.610861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15062ad-0e7f-421d-a8d1-6daf4f5d5f99 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:59.625402 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15062ad-0e7f-421d-a8d1-6daf4f5d5f99 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:59.630597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8034c959-c871-41ee-8bed-598eee1e9ecc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:59.631465 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8034c959-c871-41ee-8bed-598eee1e9ecc admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:59.631967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8034c959-c871-41ee-8bed-598eee1e9ecc admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:59.632373 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8034c959-c871-41ee-8bed-598eee1e9ecc admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:59.633139 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8034c959-c871-41ee-8bed-598eee1e9ecc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:59.634918 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8034c959-c871-41ee-8bed-598eee1e9ecc admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:59.661010 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 676/1340] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:55:59 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:55:59.669206 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434f6742-211c-4c35-adc6-9005e4ecf1e1 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:59.669449 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434f6742-211c-4c35-adc6-9005e4ecf1e1 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:59.669692 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434f6742-211c-4c35-adc6-9005e4ecf1e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:59.695775 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-b4980315-614c-49f8-86ed-e3e72d0c70f3 None None] MFA Rules not processed for user `1c0e5bd11ac14259ad376ce54bcb7025`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:59.699095 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b4980315-614c-49f8-86ed-e3e72d0c70f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:59.731610 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-434f6742-211c-4c35-adc6-9005e4ecf1e1 None None] MFA Rules not processed for user `c92f7b4d09d74b8d93d739c1938694e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:55:59.733622 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-434f6742-211c-4c35-adc6-9005e4ecf1e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:59.773347 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:55:59.833295 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 665/1341] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:59 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:59.835794 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4980315-614c-49f8-86ed-e3e72d0c70f3 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:59.861985 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4980315-614c-49f8-86ed-e3e72d0c70f3 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:55:59.869627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe85238-8d76-4749-a11c-c669e30d9aa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:55:59.870394 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe85238-8d76-4749-a11c-c669e30d9aa8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:59.870753 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe85238-8d76-4749-a11c-c669e30d9aa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:59.871163 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe85238-8d76-4749-a11c-c669e30d9aa8 admin admin] PATH_INFO: `/v3/projects/b8fcaf2d9d974aec9c96d61104709ad0/users/66e1844155924e5eac91d36bb348bd54/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:55:59.904891 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 677/1342] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:59 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:55:59.917344 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434f6742-211c-4c35-adc6-9005e4ecf1e1 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:55:59.934200 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-434f6742-211c-4c35-adc6-9005e4ecf1e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:55:59.937514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe85238-8d76-4749-a11c-c669e30d9aa8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8fcaf2d9d974aec9c96d61104709ad0, user_id=66e1844155924e5eac91d36bb348bd54, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:55:59.946015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe85238-8d76-4749-a11c-c669e30d9aa8 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:55:59.984373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 666/1343] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:55:59 2022] PUT /identity/v3/projects/b8fcaf2d9d974aec9c96d61104709ad0/users/66e1844155924e5eac91d36bb348bd54/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:55:59.988569 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0378a7f4-974f-4ae8-b9c7-79ba5471a16a None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:55:59.989352 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0378a7f4-974f-4ae8-b9c7-79ba5471a16a None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:55:59.990451 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0378a7f4-974f-4ae8-b9c7-79ba5471a16a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:00.038261 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-0378a7f4-974f-4ae8-b9c7-79ba5471a16a None None] MFA Rules not processed for user `c92f7b4d09d74b8d93d739c1938694e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:00.039557 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-0378a7f4-974f-4ae8-b9c7-79ba5471a16a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:00.168802 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 667/1344] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:55:59 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:00.170130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dfaf24-aa2a-4819-9362-fa1b4ed9a583 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:00.170410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dfaf24-aa2a-4819-9362-fa1b4ed9a583 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:00.170673 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dfaf24-aa2a-4819-9362-fa1b4ed9a583 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:00.173982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434f6742-211c-4c35-adc6-9005e4ecf1e1 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:00.260884 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-78dfaf24-aa2a-4819-9362-fa1b4ed9a583 None None] MFA Rules not processed for user `66e1844155924e5eac91d36bb348bd54`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:00.262125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-78dfaf24-aa2a-4819-9362-fa1b4ed9a583 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:00.374851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7930a41e-c230-47ec-b5f0-82694cb0e7c6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:00.376628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7930a41e-c230-47ec-b5f0-82694cb0e7c6 service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:00.377185 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7930a41e-c230-47ec-b5f0-82694cb0e7c6 service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:00.384382 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7930a41e-c230-47ec-b5f0-82694cb0e7c6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:00.417733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:00.463940 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 668/1345] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:00 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 292 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:00.463940 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dfaf24-aa2a-4819-9362-fa1b4ed9a583 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:00.504143 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dfaf24-aa2a-4819-9362-fa1b4ed9a583 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:00.527865 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d93cff-32f8-4a53-b630-cc2324fdeee0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:00.529075 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d93cff-32f8-4a53-b630-cc2324fdeee0 service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:00.529075 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d93cff-32f8-4a53-b630-cc2324fdeee0 service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:00.529312 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d93cff-32f8-4a53-b630-cc2324fdeee0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:00.559315 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7930a41e-c230-47ec-b5f0-82694cb0e7c6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:00.562670 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7930a41e-c230-47ec-b5f0-82694cb0e7c6 service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:00.563923 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d93cff-32f8-4a53-b630-cc2324fdeee0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:00.571413 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d93cff-32f8-4a53-b630-cc2324fdeee0 service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:00.582091 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 669/1346] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:00 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:00.592941 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 678/1347] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:55:59 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 688 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:00.593764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d93cff-32f8-4a53-b630-cc2324fdeee0 service nova] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:00.596652 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae8bef5-5f9a-4b39-833b-0cd0b7200dd3 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:00.599063 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae8bef5-5f9a-4b39-833b-0cd0b7200dd3 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:00.599418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae8bef5-5f9a-4b39-833b-0cd0b7200dd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:00.683086 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-bae8bef5-5f9a-4b39-833b-0cd0b7200dd3 None None] MFA Rules not processed for user `15e603f9898a430d8a53f21818e97722`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:00.685674 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-bae8bef5-5f9a-4b39-833b-0cd0b7200dd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:00.741840 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:00.808320 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d93cff-32f8-4a53-b630-cc2324fdeee0 service nova] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:00.826346 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdc4178-999e-4cc9-9286-4e59a9ff700c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:00.828686 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdc4178-999e-4cc9-9286-4e59a9ff700c admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:00.829949 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdc4178-999e-4cc9-9286-4e59a9ff700c admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:00.830224 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdc4178-999e-4cc9-9286-4e59a9ff700c admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:00.830870 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fdc4178-999e-4cc9-9286-4e59a9ff700c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:00.832856 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fdc4178-999e-4cc9-9286-4e59a9ff700c admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:00.878523 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 679/1348] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:00 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 285 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:00.908438 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 670/1349] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:56:00 2022] POST /identity/v3/projects => generated 383 bytes in 326 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:56:00.909558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae8bef5-5f9a-4b39-833b-0cd0b7200dd3 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:00.942950 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae8bef5-5f9a-4b39-833b-0cd0b7200dd3 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:00.960502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdc4178-999e-4cc9-9286-4e59a9ff700c admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:01.161493 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a731bdbb-ab9d-4205-90aa-9437a76d7b92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:01.165243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a731bdbb-ab9d-4205-90aa-9437a76d7b92 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:01.165805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a731bdbb-ab9d-4205-90aa-9437a76d7b92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:01.166240 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a731bdbb-ab9d-4205-90aa-9437a76d7b92 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:01.168405 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a731bdbb-ab9d-4205-90aa-9437a76d7b92 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:01.173580 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a731bdbb-ab9d-4205-90aa-9437a76d7b92 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:01.235066 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdc4178-999e-4cc9-9286-4e59a9ff700c admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:01.239458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 680/1350] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:56:00 2022] POST /identity/v3/users => generated 298 bytes in 330 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:56:01.243848 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbd64f2-14a6-470c-9e75-5386a775df2f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:01.244191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a731bdbb-ab9d-4205-90aa-9437a76d7b92 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:01.244490 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbd64f2-14a6-470c-9e75-5386a775df2f service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:01.244763 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbd64f2-14a6-470c-9e75-5386a775df2f service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:01.245306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbd64f2-14a6-470c-9e75-5386a775df2f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:01.267157 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a731bdbb-ab9d-4205-90aa-9437a76d7b92 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:01.286486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a46728-9da1-4550-9205-96d870225ea5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:01.287192 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a46728-9da1-4550-9205-96d870225ea5 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:01.292098 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a46728-9da1-4550-9205-96d870225ea5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:01.292431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a46728-9da1-4550-9205-96d870225ea5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:01.293499 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a46728-9da1-4550-9205-96d870225ea5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:01.296555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a46728-9da1-4550-9205-96d870225ea5 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:01.345380 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 681/1351] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:01 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:01.346409 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a46728-9da1-4550-9205-96d870225ea5 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:01.361788 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a46728-9da1-4550-9205-96d870225ea5 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:01.371800 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133d28d7-0c68-467b-a122-abef7ed9688f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:01.372511 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133d28d7-0c68-467b-a122-abef7ed9688f admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:01.372781 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133d28d7-0c68-467b-a122-abef7ed9688f admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:01.373052 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133d28d7-0c68-467b-a122-abef7ed9688f admin admin] PATH_INFO: `/v3/projects/1c7a293871eb451cbdcb27b76b27c09c/users/1c00a9e651e24bb1a2470dadbbe45481/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:01.390912 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbd64f2-14a6-470c-9e75-5386a775df2f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:01.391322 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133d28d7-0c68-467b-a122-abef7ed9688f admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c7a293871eb451cbdcb27b76b27c09c, user_id=1c00a9e651e24bb1a2470dadbbe45481, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:01.395588 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133d28d7-0c68-467b-a122-abef7ed9688f admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:01.395841 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbd64f2-14a6-470c-9e75-5386a775df2f service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:01.416437 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:01.426508 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 671/1352] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:00 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 462 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 00:56:01.434498 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 682/1353] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:56:01 2022] PUT /identity/v3/projects/1c7a293871eb451cbdcb27b76b27c09c/users/1c00a9e651e24bb1a2470dadbbe45481/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:01.439230 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbd64f2-14a6-470c-9e75-5386a775df2f service nova] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:01.454939 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbd64f2-14a6-470c-9e75-5386a775df2f service nova] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:01.613957 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a52847-4051-453e-8d00-a5e08b9bc9d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:01.615848 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a52847-4051-453e-8d00-a5e08b9bc9d2 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:01.616446 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a52847-4051-453e-8d00-a5e08b9bc9d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:01.616570 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a52847-4051-453e-8d00-a5e08b9bc9d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:01.617207 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a52847-4051-453e-8d00-a5e08b9bc9d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:01.620635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a52847-4051-453e-8d00-a5e08b9bc9d2 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:01.630299 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:01.638283 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 672/1354] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:01 2022] GET /identity/v3/roles => generated 1420 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:01.642149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133d28d7-0c68-467b-a122-abef7ed9688f admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:01.656513 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133d28d7-0c68-467b-a122-abef7ed9688f admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:01.662820 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abfd6a5-79b8-4202-bc5a-4febb3e18850 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:01.663521 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abfd6a5-79b8-4202-bc5a-4febb3e18850 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:01.663886 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abfd6a5-79b8-4202-bc5a-4febb3e18850 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:01.664202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abfd6a5-79b8-4202-bc5a-4febb3e18850 admin admin] PATH_INFO: `/v3/projects/1c7a293871eb451cbdcb27b76b27c09c/users/1c00a9e651e24bb1a2470dadbbe45481/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:01.666206 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abfd6a5-79b8-4202-bc5a-4febb3e18850 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c7a293871eb451cbdcb27b76b27c09c, user_id=1c00a9e651e24bb1a2470dadbbe45481, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:01.669376 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abfd6a5-79b8-4202-bc5a-4febb3e18850 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:01.736298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 683/1355] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:56:01 2022] PUT /identity/v3/projects/1c7a293871eb451cbdcb27b76b27c09c/users/1c00a9e651e24bb1a2470dadbbe45481/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 00:56:01.747824 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d903c2a-9f78-480f-8648-f0fbbbeb43a5 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:01.748333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d903c2a-9f78-480f-8648-f0fbbbeb43a5 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:01.748565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d903c2a-9f78-480f-8648-f0fbbbeb43a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:01.822689 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-2d903c2a-9f78-480f-8648-f0fbbbeb43a5 None None] MFA Rules not processed for user `1c00a9e651e24bb1a2470dadbbe45481`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:01.826278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-2d903c2a-9f78-480f-8648-f0fbbbeb43a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:01.931650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d45ea46-5338-428e-bae9-00bfd529e56f None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:01.931650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d45ea46-5338-428e-bae9-00bfd529e56f None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:01.931650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d45ea46-5338-428e-bae9-00bfd529e56f None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:01.936176 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 684/1356] 10.222.0.16 () {54 vars in 912 bytes} [Sun Oct 30 00:56:01 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:56:01.942357 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8e8e18-2ded-475f-afbb-29162825565c None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:01.942708 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8e8e18-2ded-475f-afbb-29162825565c None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:01.942896 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8e8e18-2ded-475f-afbb-29162825565c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:02.044366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 673/1357] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:01 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 298 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:02.055167 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-aa8e8e18-2ded-475f-afbb-29162825565c None None] MFA Rules not processed for user `81999329edfb4d4cac567d88d1f52909`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:02.056076 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d903c2a-9f78-480f-8648-f0fbbbeb43a5 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:02.056597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-aa8e8e18-2ded-475f-afbb-29162825565c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:02.057753 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-2d903c2a-9f78-480f-8648-f0fbbbeb43a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:02.216688 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d903c2a-9f78-480f-8648-f0fbbbeb43a5 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:02.228882 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799389ac-a899-46ee-9636-a065cf045e7d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:02.230818 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 685/1358] 10.222.0.16 () {58 vars in 1009 bytes} [Sun Oct 30 00:56:01 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 289 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Oct 30 00:56:02.232150 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799389ac-a899-46ee-9636-a065cf045e7d service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:02.234303 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799389ac-a899-46ee-9636-a065cf045e7d service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:02.234661 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799389ac-a899-46ee-9636-a065cf045e7d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:02.248984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8e8e18-2ded-475f-afbb-29162825565c None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:02.264894 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799389ac-a899-46ee-9636-a065cf045e7d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:02.265786 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-aa8e8e18-2ded-475f-afbb-29162825565c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:02.266940 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799389ac-a899-46ee-9636-a065cf045e7d service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:02.297186 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 674/1359] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:02 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 242 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:02.314296 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8e8e18-2ded-475f-afbb-29162825565c None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:02.319505 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef41289a-1f0d-46a9-8ef6-a924728c5099 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:02.320666 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef41289a-1f0d-46a9-8ef6-a924728c5099 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:02.320894 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef41289a-1f0d-46a9-8ef6-a924728c5099 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:02.321135 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef41289a-1f0d-46a9-8ef6-a924728c5099 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:02.329935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef41289a-1f0d-46a9-8ef6-a924728c5099 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:02.331483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef41289a-1f0d-46a9-8ef6-a924728c5099 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:02.340944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 686/1360] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:02 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:02.433299 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799389ac-a899-46ee-9636-a065cf045e7d service cinder] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:02.445931 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799389ac-a899-46ee-9636-a065cf045e7d service cinder] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:02.460894 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da2745b-e809-4635-a8f5-2dcc978051db service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:02.461640 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da2745b-e809-4635-a8f5-2dcc978051db service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:02.462045 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da2745b-e809-4635-a8f5-2dcc978051db service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:02.462416 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da2745b-e809-4635-a8f5-2dcc978051db service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:02.600769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da2745b-e809-4635-a8f5-2dcc978051db service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:02.603235 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da2745b-e809-4635-a8f5-2dcc978051db service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:02.624923 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 675/1361] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:02 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:03.177004 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc23800-19e2-497f-b238-760cfb15ac05 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:03.177691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc23800-19e2-497f-b238-760cfb15ac05 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:03.177691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc23800-19e2-497f-b238-760cfb15ac05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:03.225107 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-efc23800-19e2-497f-b238-760cfb15ac05 None None] MFA Rules not processed for user `ecaf8ec105c04b919256f852da859eba`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:03.229958 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-efc23800-19e2-497f-b238-760cfb15ac05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:03.498904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 687/1362] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:03 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 323 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:03.626026 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da2745b-e809-4635-a8f5-2dcc978051db service cinder] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:03.628599 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-8da2745b-e809-4635-a8f5-2dcc978051db service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:03.717061 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:03.801411 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da2745b-e809-4635-a8f5-2dcc978051db service cinder] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:03.809527 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07cda1e-634b-459c-b2c6-a4290e29bc42 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '63cc035057064805a95e9f12714ebade', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:03.810186 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07cda1e-634b-459c-b2c6-a4290e29bc42 service placement] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:03.810536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07cda1e-634b-459c-b2c6-a4290e29bc42 service placement] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:03.810814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07cda1e-634b-459c-b2c6-a4290e29bc42 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:03.813321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-f07cda1e-634b-459c-b2c6-a4290e29bc42 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:03.830737 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07cda1e-634b-459c-b2c6-a4290e29bc42 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:03.831788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07cda1e-634b-459c-b2c6-a4290e29bc42 service placement] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:03.844917 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 676/1363] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:03 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 219 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:06.245588 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc23800-19e2-497f-b238-760cfb15ac05 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:06.245588 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-efc23800-19e2-497f-b238-760cfb15ac05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:06.396158 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc23800-19e2-497f-b238-760cfb15ac05 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:06.414645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca469d4-f4a8-4c9d-9bb9-58f8eb05267f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:06.418961 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca469d4-f4a8-4c9d-9bb9-58f8eb05267f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:06.419315 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca469d4-f4a8-4c9d-9bb9-58f8eb05267f admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:06.419572 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca469d4-f4a8-4c9d-9bb9-58f8eb05267f admin admin] PATH_INFO: `/v3/users/0038d0604f1942fdb1fcf91140d5197d` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:06.425768 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca469d4-f4a8-4c9d-9bb9-58f8eb05267f admin admin] RBAC: Authorizing `identity:delete_user(user_id=0038d0604f1942fdb1fcf91140d5197d)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:06.426811 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: WARNING py.warnings [None req-8ca469d4-f4a8-4c9d-9bb9-58f8eb05267f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 30 00:56:06.426811 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: warnings.warn(msg) Oct 30 00:56:06.426811 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: Oct 30 00:56:06.436751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca469d4-f4a8-4c9d-9bb9-58f8eb05267f admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:06.486070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:06.661347 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c9f3c3-cbe2-436c-80f1-eefd950c5cf3 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:06.661918 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c9f3c3-cbe2-436c-80f1-eefd950c5cf3 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:06.662334 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c9f3c3-cbe2-436c-80f1-eefd950c5cf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:06.669903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8ca469d4-f4a8-4c9d-9bb9-58f8eb05267f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0038d0604f1942fdb1fcf91140d5197d'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:06.686740 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8ca469d4-f4a8-4c9d-9bb9-58f8eb05267f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0038d0604f1942fdb1fcf91140d5197d'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:06.688121 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8ca469d4-f4a8-4c9d-9bb9-58f8eb05267f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0038d0604f1942fdb1fcf91140d5197d'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:06.743579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-39c9f3c3-cbe2-436c-80f1-eefd950c5cf3 None None] MFA Rules not processed for user `15e603f9898a430d8a53f21818e97722`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:06.751686 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-39c9f3c3-cbe2-436c-80f1-eefd950c5cf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:06.764270 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8ca469d4-f4a8-4c9d-9bb9-58f8eb05267f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0038d0604f1942fdb1fcf91140d5197d'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:06.790148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8ca469d4-f4a8-4c9d-9bb9-58f8eb05267f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0038d0604f1942fdb1fcf91140d5197d'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:06.829412 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 688/1364] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:56:06 2022] DELETE /identity/v3/users/0038d0604f1942fdb1fcf91140d5197d => generated 0 bytes in 586 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:06.831389 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca469d4-f4a8-4c9d-9bb9-58f8eb05267f admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:06.835574 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-8ca469d4-f4a8-4c9d-9bb9-58f8eb05267f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:06.931627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 677/1365] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:06 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 271 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:06.933265 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa1c31a-7189-4b6e-8bfc-94d412853835 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:06.936078 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa1c31a-7189-4b6e-8bfc-94d412853835 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:06.936712 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa1c31a-7189-4b6e-8bfc-94d412853835 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:07.030508 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-3aa1c31a-7189-4b6e-8bfc-94d412853835 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:07.031697 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-3aa1c31a-7189-4b6e-8bfc-94d412853835 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:07.074491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca469d4-f4a8-4c9d-9bb9-58f8eb05267f admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:07.081010 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacd18fa-c03c-4463-b4b2-14c105ccbdba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:07.082592 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacd18fa-c03c-4463-b4b2-14c105ccbdba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:07.088140 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 678/1366] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:07.095757 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacd18fa-c03c-4463-b4b2-14c105ccbdba admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:07.097655 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacd18fa-c03c-4463-b4b2-14c105ccbdba admin admin] PATH_INFO: `/v3/users/66e1844155924e5eac91d36bb348bd54` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:07.100537 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bacd18fa-c03c-4463-b4b2-14c105ccbdba admin admin] RBAC: Authorizing `identity:delete_user(user_id=66e1844155924e5eac91d36bb348bd54)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:07.107995 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bacd18fa-c03c-4463-b4b2-14c105ccbdba admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:07.134832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa1c31a-7189-4b6e-8bfc-94d412853835 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:07.141541 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-3aa1c31a-7189-4b6e-8bfc-94d412853835 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:07.216839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-bacd18fa-c03c-4463-b4b2-14c105ccbdba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '66e1844155924e5eac91d36bb348bd54'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:07.238868 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-bacd18fa-c03c-4463-b4b2-14c105ccbdba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '66e1844155924e5eac91d36bb348bd54'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:07.239898 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-bacd18fa-c03c-4463-b4b2-14c105ccbdba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '66e1844155924e5eac91d36bb348bd54'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:07.282953 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-bacd18fa-c03c-4463-b4b2-14c105ccbdba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '66e1844155924e5eac91d36bb348bd54'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:07.305738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-bacd18fa-c03c-4463-b4b2-14c105ccbdba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '66e1844155924e5eac91d36bb348bd54'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:07.314126 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 689/1367] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:56:06 2022] DELETE /identity/v3/users/66e1844155924e5eac91d36bb348bd54 => generated 0 bytes in 484 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:07.320882 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac93990-5714-4488-8e8c-a056a1e1ead1 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:07.321528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac93990-5714-4488-8e8c-a056a1e1ead1 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:07.322057 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac93990-5714-4488-8e8c-a056a1e1ead1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:07.360133 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa1c31a-7189-4b6e-8bfc-94d412853835 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:07.366768 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-eac93990-5714-4488-8e8c-a056a1e1ead1 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:07.369586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-eac93990-5714-4488-8e8c-a056a1e1ead1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:07.422037 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:07.641866 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf13083-2e45-48c4-b785-50c2fd48b0ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:07.647626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf13083-2e45-48c4-b785-50c2fd48b0ab service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:07.648858 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf13083-2e45-48c4-b785-50c2fd48b0ab service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:07.650505 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf13083-2e45-48c4-b785-50c2fd48b0ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:07.670774 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-4bf13083-2e45-48c4-b785-50c2fd48b0ab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:07.759896 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 690/1368] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 438 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:56:07.761569 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26072d57-839d-4b8d-b2ed-97dba3a9da80 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:07.761853 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26072d57-839d-4b8d-b2ed-97dba3a9da80 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:07.762128 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26072d57-839d-4b8d-b2ed-97dba3a9da80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:07.777508 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:07.879133 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-26072d57-839d-4b8d-b2ed-97dba3a9da80 None None] MFA Rules not processed for user `1c0e5bd11ac14259ad376ce54bcb7025`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:07.880680 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-26072d57-839d-4b8d-b2ed-97dba3a9da80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:07.997866 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf13083-2e45-48c4-b785-50c2fd48b0ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:08.010481 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf13083-2e45-48c4-b785-50c2fd48b0ab service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:08.055433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 679/1369] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 923 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:08.059166 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf13083-2e45-48c4-b785-50c2fd48b0ab service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:08.066162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-4bf13083-2e45-48c4-b785-50c2fd48b0ab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:08.107285 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf13083-2e45-48c4-b785-50c2fd48b0ab service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:08.125125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f3cb58-ca4d-4ad8-bad5-f30b145bbd0c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:08.131567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f3cb58-ca4d-4ad8-bad5-f30b145bbd0c service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:08.131699 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f3cb58-ca4d-4ad8-bad5-f30b145bbd0c service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:08.134300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f3cb58-ca4d-4ad8-bad5-f30b145bbd0c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:08.147060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 691/1370] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:07 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 387 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:08.164830 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f3cb58-ca4d-4ad8-bad5-f30b145bbd0c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:08.166621 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f3cb58-ca4d-4ad8-bad5-f30b145bbd0c service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:08.203227 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 680/1371] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:08.494423 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26072d57-839d-4b8d-b2ed-97dba3a9da80 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:08.497837 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-26072d57-839d-4b8d-b2ed-97dba3a9da80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:08.521641 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26072d57-839d-4b8d-b2ed-97dba3a9da80 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:08.534105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c938c5-763b-46f3-834b-bbce1342df4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:08.534776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c938c5-763b-46f3-834b-bbce1342df4f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:08.535212 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c938c5-763b-46f3-834b-bbce1342df4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:08.535653 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c938c5-763b-46f3-834b-bbce1342df4f admin admin] PATH_INFO: `/v3/users/874f7fbc347c4cd69a2ed9c0b18d391e` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:08.538559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c938c5-763b-46f3-834b-bbce1342df4f admin admin] RBAC: Authorizing `identity:delete_user(user_id=874f7fbc347c4cd69a2ed9c0b18d391e)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:08.544764 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c938c5-763b-46f3-834b-bbce1342df4f admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:08.652173 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-d4c938c5-763b-46f3-834b-bbce1342df4f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '874f7fbc347c4cd69a2ed9c0b18d391e'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:08.664642 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-d4c938c5-763b-46f3-834b-bbce1342df4f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '874f7fbc347c4cd69a2ed9c0b18d391e'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:08.665645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-d4c938c5-763b-46f3-834b-bbce1342df4f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '874f7fbc347c4cd69a2ed9c0b18d391e'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:08.691883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-d4c938c5-763b-46f3-834b-bbce1342df4f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '874f7fbc347c4cd69a2ed9c0b18d391e'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:08.723285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-d4c938c5-763b-46f3-834b-bbce1342df4f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '874f7fbc347c4cd69a2ed9c0b18d391e'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:08.741693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 692/1372] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:56:08 2022] DELETE /identity/v3/users/874f7fbc347c4cd69a2ed9c0b18d391e => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:08.746557 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757f6be1-c943-4d9e-b26a-e544b0134726 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:08.746986 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757f6be1-c943-4d9e-b26a-e544b0134726 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:08.747148 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757f6be1-c943-4d9e-b26a-e544b0134726 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:08.783318 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-757f6be1-c943-4d9e-b26a-e544b0134726 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:08.785797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-757f6be1-c943-4d9e-b26a-e544b0134726 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:08.891749 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 681/1373] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:08.971236 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5fde99-84ba-41ce-9180-a73d0a3958ff None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:08.971537 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5fde99-84ba-41ce-9180-a73d0a3958ff None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:08.971826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5fde99-84ba-41ce-9180-a73d0a3958ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:09.009878 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757f6be1-c943-4d9e-b26a-e544b0134726 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:09.011646 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-757f6be1-c943-4d9e-b26a-e544b0134726 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:09.088473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:09.105812 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-4d5fde99-84ba-41ce-9180-a73d0a3958ff None None] MFA Rules not processed for user `4716e7afe2784e2a9b872996d895c374`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:09.107615 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-4d5fde99-84ba-41ce-9180-a73d0a3958ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:09.207434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757f6be1-c943-4d9e-b26a-e544b0134726 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:09.231069 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d84548-6844-4a0c-831c-8250ff7c874a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:09.233072 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d84548-6844-4a0c-831c-8250ff7c874a service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:09.237034 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d84548-6844-4a0c-831c-8250ff7c874a service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:09.237356 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d84548-6844-4a0c-831c-8250ff7c874a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:09.274334 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d84548-6844-4a0c-831c-8250ff7c874a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:09.275864 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d84548-6844-4a0c-831c-8250ff7c874a service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:09.298030 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 682/1374] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:09.298890 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d84548-6844-4a0c-831c-8250ff7c874a service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:09.300758 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-25d84548-6844-4a0c-831c-8250ff7c874a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:09.326584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 693/1375] 10.222.0.16 () {58 vars in 1009 bytes} [Sun Oct 30 00:56:08 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 356 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:09.352737 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d84548-6844-4a0c-831c-8250ff7c874a service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:09.362048 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5faff53-a3d3-4f91-abe4-f8191b1094b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:09.375157 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5faff53-a3d3-4f91-abe4-f8191b1094b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:09.375468 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5faff53-a3d3-4f91-abe4-f8191b1094b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:09.375756 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5faff53-a3d3-4f91-abe4-f8191b1094b0 admin admin] PATH_INFO: `/v3/projects/b8fcaf2d9d974aec9c96d61104709ad0` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:09.377224 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5faff53-a3d3-4f91-abe4-f8191b1094b0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b8fcaf2d9d974aec9c96d61104709ad0)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:09.378905 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: WARNING py.warnings [None req-f5faff53-a3d3-4f91-abe4-f8191b1094b0 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 30 00:56:09.378905 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: warnings.warn(msg) Oct 30 00:56:09.378905 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: Oct 30 00:56:09.379886 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5faff53-a3d3-4f91-abe4-f8191b1094b0 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:09.567111 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:09.600752 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-f5faff53-a3d3-4f91-abe4-f8191b1094b0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b8fcaf2d9d974aec9c96d61104709ad0', 'request_id': 'req-f5faff53-a3d3-4f91-abe4-f8191b1094b0'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:09.622317 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-f5faff53-a3d3-4f91-abe4-f8191b1094b0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b8fcaf2d9d974aec9c96d61104709ad0', 'request_id': 'req-f5faff53-a3d3-4f91-abe4-f8191b1094b0'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:09.639133 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5fde99-84ba-41ce-9180-a73d0a3958ff None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:09.640685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-4d5fde99-84ba-41ce-9180-a73d0a3958ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:09.647445 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-f5faff53-a3d3-4f91-abe4-f8191b1094b0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b8fcaf2d9d974aec9c96d61104709ad0', 'request_id': 'req-f5faff53-a3d3-4f91-abe4-f8191b1094b0'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:09.703754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-f5faff53-a3d3-4f91-abe4-f8191b1094b0 admin admin] The token cache is being invalidate because project b8fcaf2d9d974aec9c96d61104709ad0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:56:09.704178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-f5faff53-a3d3-4f91-abe4-f8191b1094b0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:09.706191 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-f5faff53-a3d3-4f91-abe4-f8191b1094b0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:09.712265 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 683/1376] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:56:09 2022] DELETE /identity/v3/projects/b8fcaf2d9d974aec9c96d61104709ad0 => generated 0 bytes in 413 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:09.717133 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5faff53-a3d3-4f91-abe4-f8191b1094b0 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:09.718839 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-f5faff53-a3d3-4f91-abe4-f8191b1094b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:09.819024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5fde99-84ba-41ce-9180-a73d0a3958ff None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:09.872766 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5faff53-a3d3-4f91-abe4-f8191b1094b0 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:09.886637 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f004a0-711e-42fc-89af-504aaba2d646 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:09.887515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f004a0-711e-42fc-89af-504aaba2d646 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:09.887817 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f004a0-711e-42fc-89af-504aaba2d646 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:09.888103 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f004a0-711e-42fc-89af-504aaba2d646 admin admin] PATH_INFO: `/v3/users/1c00a9e651e24bb1a2470dadbbe45481` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:09.892842 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f004a0-711e-42fc-89af-504aaba2d646 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1c00a9e651e24bb1a2470dadbbe45481)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:09.894891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f004a0-711e-42fc-89af-504aaba2d646 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:09.923116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb77bf96-781c-481c-8f3a-f036a2d807e7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '67f6eff8c0254cc6a210101dd7af988d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:09.923754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb77bf96-781c-481c-8f3a-f036a2d807e7 service swift] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:09.924082 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb77bf96-781c-481c-8f3a-f036a2d807e7 service swift] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:09.924546 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb77bf96-781c-481c-8f3a-f036a2d807e7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:09.927779 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-fb77bf96-781c-481c-8f3a-f036a2d807e7 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:09.979669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:09.984959 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-17f004a0-711e-42fc-89af-504aaba2d646 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1c00a9e651e24bb1a2470dadbbe45481'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:10.003494 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-17f004a0-711e-42fc-89af-504aaba2d646 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1c00a9e651e24bb1a2470dadbbe45481'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:10.004232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-17f004a0-711e-42fc-89af-504aaba2d646 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1c00a9e651e24bb1a2470dadbbe45481'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:10.013854 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-17f004a0-711e-42fc-89af-504aaba2d646 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1c00a9e651e24bb1a2470dadbbe45481'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:10.039024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-17f004a0-711e-42fc-89af-504aaba2d646 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1c00a9e651e24bb1a2470dadbbe45481'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:10.069217 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 684/1377] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:56:09 2022] DELETE /identity/v3/users/1c00a9e651e24bb1a2470dadbbe45481 => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 00:56:10.082056 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb77bf96-781c-481c-8f3a-f036a2d807e7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:10.090882 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb77bf96-781c-481c-8f3a-f036a2d807e7 service swift] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:10.131528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 694/1378] 10.222.0.16 () {60 vars in 1384 bytes} [Sun Oct 30 00:56:09 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 493 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 30 00:56:10.890669 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f004a0-711e-42fc-89af-504aaba2d646 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:10.892582 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-17f004a0-711e-42fc-89af-504aaba2d646 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:10.923775 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb204246-07a1-47b9-bc5d-6385d420fa6b None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:10.924244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb204246-07a1-47b9-bc5d-6385d420fa6b None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:10.924513 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb204246-07a1-47b9-bc5d-6385d420fa6b None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:10.926178 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 695/1379] 10.222.0.16 () {54 vars in 912 bytes} [Sun Oct 30 00:56:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:56:10.934976 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e60bcf2-91ef-4718-8891-f01de1f8f729 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:10.935273 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e60bcf2-91ef-4718-8891-f01de1f8f729 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:10.935514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e60bcf2-91ef-4718-8891-f01de1f8f729 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:10.973506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-7e60bcf2-91ef-4718-8891-f01de1f8f729 None None] MFA Rules not processed for user `81999329edfb4d4cac567d88d1f52909`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:10.974818 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-7e60bcf2-91ef-4718-8891-f01de1f8f729 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:11.112322 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:11.132427 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 696/1380] 10.222.0.16 () {58 vars in 1009 bytes} [Sun Oct 30 00:56:10 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:56:11.162563 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e60bcf2-91ef-4718-8891-f01de1f8f729 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:11.171211 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-7e60bcf2-91ef-4718-8891-f01de1f8f729 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:11.187667 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e60bcf2-91ef-4718-8891-f01de1f8f729 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:11.198829 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7686b46b-b545-4b56-81e9-8487d405cad8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:11.199752 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7686b46b-b545-4b56-81e9-8487d405cad8 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:11.200175 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7686b46b-b545-4b56-81e9-8487d405cad8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:11.200579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7686b46b-b545-4b56-81e9-8487d405cad8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:11.212196 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f004a0-711e-42fc-89af-504aaba2d646 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:11.231223 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7686b46b-b545-4b56-81e9-8487d405cad8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:11.233072 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7686b46b-b545-4b56-81e9-8487d405cad8 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:11.235486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2ea432-0174-4512-9ae6-5801ad6649d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:11.239617 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2ea432-0174-4512-9ae6-5801ad6649d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:11.240467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2ea432-0174-4512-9ae6-5801ad6649d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:11.240637 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2ea432-0174-4512-9ae6-5801ad6649d0 admin admin] PATH_INFO: `/v3/projects/ce1df4d4d27a412b91845bc0fb02c3b9` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:11.244882 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 697/1381] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:11 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:11.254947 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2ea432-0174-4512-9ae6-5801ad6649d0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ce1df4d4d27a412b91845bc0fb02c3b9)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:11.263253 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2ea432-0174-4512-9ae6-5801ad6649d0 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:11.450640 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4c2ea432-0174-4512-9ae6-5801ad6649d0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ce1df4d4d27a412b91845bc0fb02c3b9', 'request_id': 'req-4c2ea432-0174-4512-9ae6-5801ad6649d0'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:11.467820 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4c2ea432-0174-4512-9ae6-5801ad6649d0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ce1df4d4d27a412b91845bc0fb02c3b9', 'request_id': 'req-4c2ea432-0174-4512-9ae6-5801ad6649d0'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:11.491535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4c2ea432-0174-4512-9ae6-5801ad6649d0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ce1df4d4d27a412b91845bc0fb02c3b9', 'request_id': 'req-4c2ea432-0174-4512-9ae6-5801ad6649d0'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:11.521785 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7686b46b-b545-4b56-81e9-8487d405cad8 service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:11.523113 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4c2ea432-0174-4512-9ae6-5801ad6649d0 admin admin] The token cache is being invalidate because project ce1df4d4d27a412b91845bc0fb02c3b9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:56:11.523227 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-7686b46b-b545-4b56-81e9-8487d405cad8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:11.523409 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4c2ea432-0174-4512-9ae6-5801ad6649d0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:11.524530 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4c2ea432-0174-4512-9ae6-5801ad6649d0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:11.555707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 685/1382] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:56:10 2022] DELETE /identity/v3/projects/ce1df4d4d27a412b91845bc0fb02c3b9 => generated 0 bytes in 666 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:11.556722 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2ea432-0174-4512-9ae6-5801ad6649d0 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:11.565841 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7686b46b-b545-4b56-81e9-8487d405cad8 service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:11.587749 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-4c2ea432-0174-4512-9ae6-5801ad6649d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:11.769028 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08bdd70-ad36-4f4d-ab00-d9e26bd0c937 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:11.770221 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08bdd70-ad36-4f4d-ab00-d9e26bd0c937 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:11.770697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08bdd70-ad36-4f4d-ab00-d9e26bd0c937 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:11.771221 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08bdd70-ad36-4f4d-ab00-d9e26bd0c937 admin admin] PATH_INFO: `/v3/projects/1c7a293871eb451cbdcb27b76b27c09c` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:11.773006 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b08bdd70-ad36-4f4d-ab00-d9e26bd0c937 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1c7a293871eb451cbdcb27b76b27c09c)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:11.778448 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b08bdd70-ad36-4f4d-ab00-d9e26bd0c937 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:11.806595 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2ea432-0174-4512-9ae6-5801ad6649d0 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:11.818747 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9000cdd-c431-484e-9827-9fdeeb58167b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:11.819839 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9000cdd-c431-484e-9827-9fdeeb58167b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:11.820136 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9000cdd-c431-484e-9827-9fdeeb58167b admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:11.821242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9000cdd-c431-484e-9827-9fdeeb58167b admin admin] PATH_INFO: `/v3/projects/eb71114e972447f6bcfad53c4d2bad7e` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:11.822293 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9000cdd-c431-484e-9827-9fdeeb58167b admin admin] RBAC: Authorizing `identity:delete_project(project_id=eb71114e972447f6bcfad53c4d2bad7e)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:11.839088 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9000cdd-c431-484e-9827-9fdeeb58167b admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:11.860702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b08bdd70-ad36-4f4d-ab00-d9e26bd0c937 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1c7a293871eb451cbdcb27b76b27c09c', 'request_id': 'req-b08bdd70-ad36-4f4d-ab00-d9e26bd0c937'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:11.876201 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b08bdd70-ad36-4f4d-ab00-d9e26bd0c937 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1c7a293871eb451cbdcb27b76b27c09c', 'request_id': 'req-b08bdd70-ad36-4f4d-ab00-d9e26bd0c937'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:11.887186 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:11.888662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b08bdd70-ad36-4f4d-ab00-d9e26bd0c937 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1c7a293871eb451cbdcb27b76b27c09c', 'request_id': 'req-b08bdd70-ad36-4f4d-ab00-d9e26bd0c937'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:11.907924 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b08bdd70-ad36-4f4d-ab00-d9e26bd0c937 admin admin] The token cache is being invalidate because project 1c7a293871eb451cbdcb27b76b27c09c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:56:11.908377 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b08bdd70-ad36-4f4d-ab00-d9e26bd0c937 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:11.909154 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b08bdd70-ad36-4f4d-ab00-d9e26bd0c937 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:11.916662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 698/1383] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:56:11 2022] DELETE /identity/v3/projects/1c7a293871eb451cbdcb27b76b27c09c => generated 0 bytes in 390 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:11.919503 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f227b4-cbd0-4ece-bee8-08885727ae73 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:11.920471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f227b4-cbd0-4ece-bee8-08885727ae73 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:11.921604 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f227b4-cbd0-4ece-bee8-08885727ae73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:11.957517 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-10f227b4-cbd0-4ece-bee8-08885727ae73 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:11.959110 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-10f227b4-cbd0-4ece-bee8-08885727ae73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:12.052393 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a9000cdd-c431-484e-9827-9fdeeb58167b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eb71114e972447f6bcfad53c4d2bad7e', 'request_id': 'req-a9000cdd-c431-484e-9827-9fdeeb58167b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:12.065741 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 699/1384] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:12.065741 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aa3f7e-760f-4d78-b063-091eed4d6638 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:12.065741 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aa3f7e-760f-4d78-b063-091eed4d6638 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:12.065741 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aa3f7e-760f-4d78-b063-091eed4d6638 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:12.082200 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a9000cdd-c431-484e-9827-9fdeeb58167b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eb71114e972447f6bcfad53c4d2bad7e', 'request_id': 'req-a9000cdd-c431-484e-9827-9fdeeb58167b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:12.109552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a9000cdd-c431-484e-9827-9fdeeb58167b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eb71114e972447f6bcfad53c4d2bad7e', 'request_id': 'req-a9000cdd-c431-484e-9827-9fdeeb58167b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:12.121563 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-44aa3f7e-760f-4d78-b063-091eed4d6638 None None] MFA Rules not processed for user `81999329edfb4d4cac567d88d1f52909`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:12.122247 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-44aa3f7e-760f-4d78-b063-091eed4d6638 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:12.153880 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a9000cdd-c431-484e-9827-9fdeeb58167b admin admin] The token cache is being invalidate because project eb71114e972447f6bcfad53c4d2bad7e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:56:12.154187 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a9000cdd-c431-484e-9827-9fdeeb58167b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:12.162346 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a9000cdd-c431-484e-9827-9fdeeb58167b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:12.168093 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 686/1385] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:56:11 2022] DELETE /identity/v3/projects/eb71114e972447f6bcfad53c4d2bad7e => generated 0 bytes in 612 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:12.169490 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9c4b49-360c-4dab-8f22-a05e1af208b3 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:12.169769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9c4b49-360c-4dab-8f22-a05e1af208b3 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:12.170041 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9c4b49-360c-4dab-8f22-a05e1af208b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:12.216442 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-0f9c4b49-360c-4dab-8f22-a05e1af208b3 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:12.218372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-0f9c4b49-360c-4dab-8f22-a05e1af208b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:12.266647 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:12.311640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 700/1386] 10.222.0.43 () {58 vars in 1015 bytes} [Sun Oct 30 00:56:12 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:12.315995 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44aa3f7e-760f-4d78-b063-091eed4d6638 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:12.318583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-44aa3f7e-760f-4d78-b063-091eed4d6638 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:12.449734 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 687/1387] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:12.451123 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45129def-7569-4cf4-8aa9-0f098174e3e8 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:12.451611 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45129def-7569-4cf4-8aa9-0f098174e3e8 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:12.451985 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45129def-7569-4cf4-8aa9-0f098174e3e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:12.494869 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-45129def-7569-4cf4-8aa9-0f098174e3e8 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:12.499825 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-45129def-7569-4cf4-8aa9-0f098174e3e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:12.531426 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 688/1388] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:12.541529 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45129def-7569-4cf4-8aa9-0f098174e3e8 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:12.544569 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-45129def-7569-4cf4-8aa9-0f098174e3e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:12.589499 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45129def-7569-4cf4-8aa9-0f098174e3e8 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:12.610290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f24a9a-f1a3-4085-beb5-6aaf5cdfc05b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:12.611894 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f24a9a-f1a3-4085-beb5-6aaf5cdfc05b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:12.612203 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f24a9a-f1a3-4085-beb5-6aaf5cdfc05b admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:12.615517 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f24a9a-f1a3-4085-beb5-6aaf5cdfc05b admin admin] PATH_INFO: `/v3/users/ffbdb986ac7f48168b83888558fc23d7` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:12.617786 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f24a9a-f1a3-4085-beb5-6aaf5cdfc05b admin admin] RBAC: Authorizing `identity:delete_user(user_id=ffbdb986ac7f48168b83888558fc23d7)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:12.623320 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f24a9a-f1a3-4085-beb5-6aaf5cdfc05b admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:12.667550 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44aa3f7e-760f-4d78-b063-091eed4d6638 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:12.689997 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bf99d9-5018-4ebb-b822-bc4784a40fbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:12.691315 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bf99d9-5018-4ebb-b822-bc4784a40fbb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:12.691709 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bf99d9-5018-4ebb-b822-bc4784a40fbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:12.692051 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bf99d9-5018-4ebb-b822-bc4784a40fbb admin admin] PATH_INFO: `/v3/users/a319903641df4cb69edc2f2508b544ec` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:12.701417 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bf99d9-5018-4ebb-b822-bc4784a40fbb admin admin] RBAC: Authorizing `identity:delete_user(user_id=a319903641df4cb69edc2f2508b544ec)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:12.706672 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bf99d9-5018-4ebb-b822-bc4784a40fbb admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:12.736938 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-74f24a9a-f1a3-4085-beb5-6aaf5cdfc05b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ffbdb986ac7f48168b83888558fc23d7'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:12.755113 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-74f24a9a-f1a3-4085-beb5-6aaf5cdfc05b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ffbdb986ac7f48168b83888558fc23d7'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:12.755999 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-74f24a9a-f1a3-4085-beb5-6aaf5cdfc05b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ffbdb986ac7f48168b83888558fc23d7'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:12.768963 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-74f24a9a-f1a3-4085-beb5-6aaf5cdfc05b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ffbdb986ac7f48168b83888558fc23d7'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:12.825016 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-74f24a9a-f1a3-4085-beb5-6aaf5cdfc05b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ffbdb986ac7f48168b83888558fc23d7'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:12.843771 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 689/1389] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:56:12 2022] DELETE /identity/v3/users/ffbdb986ac7f48168b83888558fc23d7 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:12.848210 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab68613-257a-43cc-b6bf-64c0bab1b3fb None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:12.848496 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab68613-257a-43cc-b6bf-64c0bab1b3fb None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:12.848772 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab68613-257a-43cc-b6bf-64c0bab1b3fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:12.866727 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:12.870891 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-e8bf99d9-5018-4ebb-b822-bc4784a40fbb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a319903641df4cb69edc2f2508b544ec'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:12.883976 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-e8bf99d9-5018-4ebb-b822-bc4784a40fbb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a319903641df4cb69edc2f2508b544ec'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:12.884790 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-e8bf99d9-5018-4ebb-b822-bc4784a40fbb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a319903641df4cb69edc2f2508b544ec'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:12.907909 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-6ab68613-257a-43cc-b6bf-64c0bab1b3fb None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:12.909095 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-6ab68613-257a-43cc-b6bf-64c0bab1b3fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:12.928500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-e8bf99d9-5018-4ebb-b822-bc4784a40fbb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a319903641df4cb69edc2f2508b544ec'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:12.963854 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-e8bf99d9-5018-4ebb-b822-bc4784a40fbb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a319903641df4cb69edc2f2508b544ec'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:12.978302 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 701/1390] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:56:12 2022] DELETE /identity/v3/users/a319903641df4cb69edc2f2508b544ec => generated 0 bytes in 666 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:12.978960 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bf99d9-5018-4ebb-b822-bc4784a40fbb admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:12.999332 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e8bf99d9-5018-4ebb-b822-bc4784a40fbb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:13.072220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 690/1391] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:13.075659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab68613-257a-43cc-b6bf-64c0bab1b3fb None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:13.079990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-6ab68613-257a-43cc-b6bf-64c0bab1b3fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:13.107784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab68613-257a-43cc-b6bf-64c0bab1b3fb None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:13.154754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:13.253376 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acdad7f5-32e3-4da6-bb40-fb45ad3656cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:13.254795 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdad7f5-32e3-4da6-bb40-fb45ad3656cc admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:13.255084 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdad7f5-32e3-4da6-bb40-fb45ad3656cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:13.255356 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdad7f5-32e3-4da6-bb40-fb45ad3656cc admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:13.258389 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acdad7f5-32e3-4da6-bb40-fb45ad3656cc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:56:13.258744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acdad7f5-32e3-4da6-bb40-fb45ad3656cc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:13.264141 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acdad7f5-32e3-4da6-bb40-fb45ad3656cc admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:13.292466 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 691/1392] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:56:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:56:13.294657 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55aebb6e-efbe-4b3e-8f19-67e9fdbff6e3 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:13.294963 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55aebb6e-efbe-4b3e-8f19-67e9fdbff6e3 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:13.295234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55aebb6e-efbe-4b3e-8f19-67e9fdbff6e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:13.352930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-55aebb6e-efbe-4b3e-8f19-67e9fdbff6e3 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:13.353891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-55aebb6e-efbe-4b3e-8f19-67e9fdbff6e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:13.408137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 692/1393] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:13.416716 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bf99d9-5018-4ebb-b822-bc4784a40fbb admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:13.416820 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55aebb6e-efbe-4b3e-8f19-67e9fdbff6e3 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:13.428423 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a517d47c-7d57-44a0-9ee0-5fe7dff7b6eb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:13.429429 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a517d47c-7d57-44a0-9ee0-5fe7dff7b6eb service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:13.429926 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a517d47c-7d57-44a0-9ee0-5fe7dff7b6eb service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:13.431234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a517d47c-7d57-44a0-9ee0-5fe7dff7b6eb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:13.433136 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-a517d47c-7d57-44a0-9ee0-5fe7dff7b6eb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:13.447842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a517d47c-7d57-44a0-9ee0-5fe7dff7b6eb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:13.451060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a517d47c-7d57-44a0-9ee0-5fe7dff7b6eb service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:13.458684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55aebb6e-efbe-4b3e-8f19-67e9fdbff6e3 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:13.476875 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212650a9-6fb6-4478-82f5-7ac2604603e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:13.478862 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212650a9-6fb6-4478-82f5-7ac2604603e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:13.479155 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212650a9-6fb6-4478-82f5-7ac2604603e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:13.480800 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212650a9-6fb6-4478-82f5-7ac2604603e4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:13.485099 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212650a9-6fb6-4478-82f5-7ac2604603e4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:56:13.487655 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212650a9-6fb6-4478-82f5-7ac2604603e4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:13.494204 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212650a9-6fb6-4478-82f5-7ac2604603e4 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:13.500093 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 702/1394] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:12 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 521 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:13.504288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a517d47c-7d57-44a0-9ee0-5fe7dff7b6eb service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:13.541857 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a517d47c-7d57-44a0-9ee0-5fe7dff7b6eb service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:13.548526 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2c2b1d-f6ab-4048-a35a-83e21518cc96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:13.549198 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2c2b1d-f6ab-4048-a35a-83e21518cc96 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:13.549458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2c2b1d-f6ab-4048-a35a-83e21518cc96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:13.549701 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2c2b1d-f6ab-4048-a35a-83e21518cc96 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:13.550835 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2c2b1d-f6ab-4048-a35a-83e21518cc96 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:13.552923 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2c2b1d-f6ab-4048-a35a-83e21518cc96 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:13.557365 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 693/1395] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:56:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:56:13.568825 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212650a9-6fb6-4478-82f5-7ac2604603e4 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:13.613612 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212650a9-6fb6-4478-82f5-7ac2604603e4 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:13.647815 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 703/1396] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:56:13 2022] POST /identity/v3/projects => generated 373 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:56:13.648402 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2995ab0c-224a-4f4c-89b3-92632eb5f83f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:13.651088 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2995ab0c-224a-4f4c-89b3-92632eb5f83f admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:13.651377 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2995ab0c-224a-4f4c-89b3-92632eb5f83f admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:13.651599 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2c2b1d-f6ab-4048-a35a-83e21518cc96 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:13.655591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2995ab0c-224a-4f4c-89b3-92632eb5f83f admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:13.656510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-8c2c2b1d-f6ab-4048-a35a-83e21518cc96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:13.662881 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2995ab0c-224a-4f4c-89b3-92632eb5f83f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:13.676982 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2995ab0c-224a-4f4c-89b3-92632eb5f83f admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:13.788670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 694/1397] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:56:13 2022] POST /identity/v3/projects => generated 363 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:56:13.792707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2995ab0c-224a-4f4c-89b3-92632eb5f83f admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:13.999215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2995ab0c-224a-4f4c-89b3-92632eb5f83f admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:14.013859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192055e6-0de4-4054-b302-4034ec569a27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:14.014639 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192055e6-0de4-4054-b302-4034ec569a27 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:14.014935 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192055e6-0de4-4054-b302-4034ec569a27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:14.015210 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192055e6-0de4-4054-b302-4034ec569a27 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:14.015862 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192055e6-0de4-4054-b302-4034ec569a27 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:14.017819 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192055e6-0de4-4054-b302-4034ec569a27 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:14.074307 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 695/1398] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:56:13 2022] POST /identity/v3/users => generated 287 bytes in 285 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:56:14.078257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192055e6-0de4-4054-b302-4034ec569a27 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:14.114114 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192055e6-0de4-4054-b302-4034ec569a27 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:14.125400 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3cb06dc-a1f6-495c-ae6c-246cc1cabe5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:14.132064 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cb06dc-a1f6-495c-ae6c-246cc1cabe5e admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:14.132064 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cb06dc-a1f6-495c-ae6c-246cc1cabe5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:14.132064 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cb06dc-a1f6-495c-ae6c-246cc1cabe5e admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:14.132064 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3cb06dc-a1f6-495c-ae6c-246cc1cabe5e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:14.132422 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2c2b1d-f6ab-4048-a35a-83e21518cc96 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:14.132878 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3cb06dc-a1f6-495c-ae6c-246cc1cabe5e admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:14.188970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 696/1399] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:56:14 2022] POST /identity/v3/users => generated 282 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:56:14.189948 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3cb06dc-a1f6-495c-ae6c-246cc1cabe5e admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:14.215397 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3cb06dc-a1f6-495c-ae6c-246cc1cabe5e admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:14.236160 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b70e63b-2b1e-4227-b46a-1591a287e557 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:14.238274 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b70e63b-2b1e-4227-b46a-1591a287e557 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:14.238556 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b70e63b-2b1e-4227-b46a-1591a287e557 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:14.238851 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b70e63b-2b1e-4227-b46a-1591a287e557 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:14.239534 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b70e63b-2b1e-4227-b46a-1591a287e557 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:14.241291 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b70e63b-2b1e-4227-b46a-1591a287e557 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:14.288023 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 697/1400] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:14 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:14.288023 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b70e63b-2b1e-4227-b46a-1591a287e557 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:14.310267 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b70e63b-2b1e-4227-b46a-1591a287e557 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:14.327319 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acf9ff3-df40-40ce-9979-0e49ddfc0264 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:14.328709 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acf9ff3-df40-40ce-9979-0e49ddfc0264 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:14.328988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acf9ff3-df40-40ce-9979-0e49ddfc0264 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:14.329257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acf9ff3-df40-40ce-9979-0e49ddfc0264 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:14.329943 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4acf9ff3-df40-40ce-9979-0e49ddfc0264 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:14.331566 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4acf9ff3-df40-40ce-9979-0e49ddfc0264 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:14.368182 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 698/1401] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:14 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:14.369576 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acf9ff3-df40-40ce-9979-0e49ddfc0264 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:14.394683 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acf9ff3-df40-40ce-9979-0e49ddfc0264 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:14.411091 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13961c38-0374-404c-8f17-5c9834bfd9cc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:14.414893 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13961c38-0374-404c-8f17-5c9834bfd9cc service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:14.416013 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909301dc-2eb3-45df-9044-41d3917c8700 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:14.416878 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909301dc-2eb3-45df-9044-41d3917c8700 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:14.418687 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909301dc-2eb3-45df-9044-41d3917c8700 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:14.419172 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909301dc-2eb3-45df-9044-41d3917c8700 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:14.421643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909301dc-2eb3-45df-9044-41d3917c8700 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:14.422978 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13961c38-0374-404c-8f17-5c9834bfd9cc service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:14.423521 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13961c38-0374-404c-8f17-5c9834bfd9cc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:14.424820 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909301dc-2eb3-45df-9044-41d3917c8700 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:14.456750 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 699/1402] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:14 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:14.458098 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909301dc-2eb3-45df-9044-41d3917c8700 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:14.479049 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13961c38-0374-404c-8f17-5c9834bfd9cc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:14.481299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13961c38-0374-404c-8f17-5c9834bfd9cc service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:14.482385 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909301dc-2eb3-45df-9044-41d3917c8700 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:14.492431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:DELETE=4,keystone:SELECT=94,keystone:INSERT=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:14.497832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16dbe23-46ab-4b3b-b783-316f9e0569fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:14.498588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16dbe23-46ab-4b3b-b783-316f9e0569fb admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:14.499043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16dbe23-46ab-4b3b-b783-316f9e0569fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:14.499398 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16dbe23-46ab-4b3b-b783-316f9e0569fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:14.500497 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a16dbe23-46ab-4b3b-b783-316f9e0569fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:14.502331 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a16dbe23-46ab-4b3b-b783-316f9e0569fb admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:14.504518 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 704/1403] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 855 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:14.506906 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13961c38-0374-404c-8f17-5c9834bfd9cc service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:14.511447 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-13961c38-0374-404c-8f17-5c9834bfd9cc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:14.522511 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:14.535793 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13961c38-0374-404c-8f17-5c9834bfd9cc service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:14.544774 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 700/1404] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:14 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:14.544960 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760149d8-aeca-4dec-b878-df1c85d3af81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:14.545086 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16dbe23-46ab-4b3b-b783-316f9e0569fb admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:14.545168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760149d8-aeca-4dec-b878-df1c85d3af81 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:14.545168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760149d8-aeca-4dec-b878-df1c85d3af81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:14.545168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760149d8-aeca-4dec-b878-df1c85d3af81 admin admin] PATH_INFO: `/v3/projects/00101170482f49fe9093143e4d5a4d05` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:14.546453 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760149d8-aeca-4dec-b878-df1c85d3af81 admin admin] RBAC: Authorizing `identity:delete_project(project_id=00101170482f49fe9093143e4d5a4d05)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:14.549276 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760149d8-aeca-4dec-b878-df1c85d3af81 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:14.582730 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16dbe23-46ab-4b3b-b783-316f9e0569fb admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:14.596698 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b778af-81e3-4b3c-b77b-9cfb14209dee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:14.597410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b778af-81e3-4b3c-b77b-9cfb14209dee admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:14.598028 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b778af-81e3-4b3c-b77b-9cfb14209dee admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:14.598300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b778af-81e3-4b3c-b77b-9cfb14209dee admin admin] PATH_INFO: `/v3/projects/782a4eade27d42298bbf4519f07d3e7f/users/391b5a6cb3bd4569ad3df5663194d1f4/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:14.626733 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b778af-81e3-4b3c-b77b-9cfb14209dee admin admin] RBAC: Authorizing `identity:create_grant(project_id=782a4eade27d42298bbf4519f07d3e7f, user_id=391b5a6cb3bd4569ad3df5663194d1f4, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:14.633149 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b778af-81e3-4b3c-b77b-9cfb14209dee admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:14.679976 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 701/1405] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:56:14 2022] PUT /identity/v3/projects/782a4eade27d42298bbf4519f07d3e7f/users/391b5a6cb3bd4569ad3df5663194d1f4/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:14.681174 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b778af-81e3-4b3c-b77b-9cfb14209dee admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:14.712323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-760149d8-aeca-4dec-b878-df1c85d3af81 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '00101170482f49fe9093143e4d5a4d05', 'request_id': 'req-760149d8-aeca-4dec-b878-df1c85d3af81'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:14.759841 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-760149d8-aeca-4dec-b878-df1c85d3af81 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '00101170482f49fe9093143e4d5a4d05', 'request_id': 'req-760149d8-aeca-4dec-b878-df1c85d3af81'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:14.791251 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-760149d8-aeca-4dec-b878-df1c85d3af81 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '00101170482f49fe9093143e4d5a4d05', 'request_id': 'req-760149d8-aeca-4dec-b878-df1c85d3af81'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:14.868349 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-760149d8-aeca-4dec-b878-df1c85d3af81 admin admin] The token cache is being invalidate because project 00101170482f49fe9093143e4d5a4d05 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:56:14.872140 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-760149d8-aeca-4dec-b878-df1c85d3af81 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:14.875375 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-760149d8-aeca-4dec-b878-df1c85d3af81 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:14.889316 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 705/1406] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:56:14 2022] DELETE /identity/v3/projects/00101170482f49fe9093143e4d5a4d05 => generated 0 bytes in 384 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:14.895242 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a7472f-69bb-4d8e-ba78-bf0d85fe8136 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:14.897988 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a7472f-69bb-4d8e-ba78-bf0d85fe8136 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:14.900531 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a7472f-69bb-4d8e-ba78-bf0d85fe8136 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:14.903681 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b778af-81e3-4b3c-b77b-9cfb14209dee admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:14.914789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48fa07c-bba8-4bfd-a156-8b138d20bd47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:14.915614 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48fa07c-bba8-4bfd-a156-8b138d20bd47 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:14.916245 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48fa07c-bba8-4bfd-a156-8b138d20bd47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:14.916632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48fa07c-bba8-4bfd-a156-8b138d20bd47 admin admin] PATH_INFO: `/v3/projects/e59bb9a40a29499f916536e02ba1c238/users/b3473287e6224664bc645a8f442c6176/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:14.967082 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48fa07c-bba8-4bfd-a156-8b138d20bd47 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e59bb9a40a29499f916536e02ba1c238, user_id=b3473287e6224664bc645a8f442c6176, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:14.969969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48fa07c-bba8-4bfd-a156-8b138d20bd47 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:14.976558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-45a7472f-69bb-4d8e-ba78-bf0d85fe8136 None None] MFA Rules not processed for user `391b5a6cb3bd4569ad3df5663194d1f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:14.978565 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-45a7472f-69bb-4d8e-ba78-bf0d85fe8136 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:15.011489 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 702/1407] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:56:14 2022] PUT /identity/v3/projects/e59bb9a40a29499f916536e02ba1c238/users/b3473287e6224664bc645a8f442c6176/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:15.016041 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1736009a-9344-4946-ac91-21a049d0fc94 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:15.021271 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1736009a-9344-4946-ac91-21a049d0fc94 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:15.021723 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1736009a-9344-4946-ac91-21a049d0fc94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:15.072537 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-1736009a-9344-4946-ac91-21a049d0fc94 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:15.073722 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-1736009a-9344-4946-ac91-21a049d0fc94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:15.153243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 706/1408] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:14 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:15.155214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4e3adb-7a59-4764-8bcb-48f413e16442 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:15.155505 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4e3adb-7a59-4764-8bcb-48f413e16442 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:15.155802 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4e3adb-7a59-4764-8bcb-48f413e16442 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:15.217577 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 703/1409] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:15.228077 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e393a299-bc87-436a-b01a-224f10576f49 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:15.228475 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e393a299-bc87-436a-b01a-224f10576f49 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:15.229980 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e393a299-bc87-436a-b01a-224f10576f49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:15.237143 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-bf4e3adb-7a59-4764-8bcb-48f413e16442 None None] MFA Rules not processed for user `b3473287e6224664bc645a8f442c6176`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:15.238419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-bf4e3adb-7a59-4764-8bcb-48f413e16442 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:15.282453 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-e393a299-bc87-436a-b01a-224f10576f49 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:15.284024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-e393a299-bc87-436a-b01a-224f10576f49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:15.304301 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 704/1410] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:56:15.305249 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e393a299-bc87-436a-b01a-224f10576f49 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:15.315234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e393a299-bc87-436a-b01a-224f10576f49 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:15.320766 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee24fd0-458c-46e0-b456-b8b1cb0fd8ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:15.321384 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee24fd0-458c-46e0-b456-b8b1cb0fd8ab admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:15.321804 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee24fd0-458c-46e0-b456-b8b1cb0fd8ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:15.322204 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee24fd0-458c-46e0-b456-b8b1cb0fd8ab admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:15.323115 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee24fd0-458c-46e0-b456-b8b1cb0fd8ab admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:56:15.323543 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee24fd0-458c-46e0-b456-b8b1cb0fd8ab admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:15.325829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee24fd0-458c-46e0-b456-b8b1cb0fd8ab admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:15.354246 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 705/1411] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:56:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:56:15.354246 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee24fd0-458c-46e0-b456-b8b1cb0fd8ab admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:15.363130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee24fd0-458c-46e0-b456-b8b1cb0fd8ab admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:15.376700 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0345f4-0f45-4867-8f71-7ce908a72382 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:15.377321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0345f4-0f45-4867-8f71-7ce908a72382 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:15.377590 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0345f4-0f45-4867-8f71-7ce908a72382 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:15.377855 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0345f4-0f45-4867-8f71-7ce908a72382 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:15.378488 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0345f4-0f45-4867-8f71-7ce908a72382 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:15.381291 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0345f4-0f45-4867-8f71-7ce908a72382 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:15.418174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 707/1412] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:15 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 263 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:15.429514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4e3adb-7a59-4764-8bcb-48f413e16442 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:15.433051 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-bf4e3adb-7a59-4764-8bcb-48f413e16442 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:15.441537 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:15.450072 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 706/1413] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:56:15 2022] POST /identity/v3/projects => generated 357 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:56:15.451094 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0345f4-0f45-4867-8f71-7ce908a72382 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:15.509487 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:15.595625 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4e3adb-7a59-4764-8bcb-48f413e16442 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:15.628249 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0345f4-0f45-4867-8f71-7ce908a72382 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:15.643310 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62df7a10-b7a8-45ac-858e-46344c56bca9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:15.644328 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62df7a10-b7a8-45ac-858e-46344c56bca9 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:15.644607 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62df7a10-b7a8-45ac-858e-46344c56bca9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:15.645191 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62df7a10-b7a8-45ac-858e-46344c56bca9 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:15.646587 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62df7a10-b7a8-45ac-858e-46344c56bca9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:15.657593 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62df7a10-b7a8-45ac-858e-46344c56bca9 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:15.715320 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 707/1414] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:56:15 2022] POST /identity/v3/users => generated 279 bytes in 265 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:56:15.716327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62df7a10-b7a8-45ac-858e-46344c56bca9 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:15.720152 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-62df7a10-b7a8-45ac-858e-46344c56bca9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:15.731570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b22061-be8d-436b-b1ba-8f1d5f1cbab5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:15.733744 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b22061-be8d-436b-b1ba-8f1d5f1cbab5 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:15.734028 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b22061-be8d-436b-b1ba-8f1d5f1cbab5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:15.734296 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b22061-be8d-436b-b1ba-8f1d5f1cbab5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:15.869459 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b22061-be8d-436b-b1ba-8f1d5f1cbab5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:15.871470 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b22061-be8d-436b-b1ba-8f1d5f1cbab5 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:15.894279 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 708/1415] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:15 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 465 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:15.896552 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b22061-be8d-436b-b1ba-8f1d5f1cbab5 service cinder] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:15.929599 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b22061-be8d-436b-b1ba-8f1d5f1cbab5 service cinder] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:15.940712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e968695-8d29-47c2-b8f1-ea70c45df06c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:15.941339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e968695-8d29-47c2-b8f1-ea70c45df06c admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:15.941614 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e968695-8d29-47c2-b8f1-ea70c45df06c admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:15.941883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e968695-8d29-47c2-b8f1-ea70c45df06c admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:15.942524 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e968695-8d29-47c2-b8f1-ea70c45df06c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:15.944174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e968695-8d29-47c2-b8f1-ea70c45df06c admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:15.946585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62df7a10-b7a8-45ac-858e-46344c56bca9 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:15.965329 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abf1b88-03ae-4cd9-b12f-614a465935ba service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:15.965956 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abf1b88-03ae-4cd9-b12f-614a465935ba service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:15.966237 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abf1b88-03ae-4cd9-b12f-614a465935ba service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:15.966507 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abf1b88-03ae-4cd9-b12f-614a465935ba service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:15.969259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 709/1416] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:15 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:15.970165 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e968695-8d29-47c2-b8f1-ea70c45df06c admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:15.998458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e968695-8d29-47c2-b8f1-ea70c45df06c admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:16.023527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b018a89-7daf-49a7-9123-189a2cdffd71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:16.024477 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b018a89-7daf-49a7-9123-189a2cdffd71 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:16.026684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b018a89-7daf-49a7-9123-189a2cdffd71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:16.027459 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b018a89-7daf-49a7-9123-189a2cdffd71 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:16.032780 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b018a89-7daf-49a7-9123-189a2cdffd71 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:16.035683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b018a89-7daf-49a7-9123-189a2cdffd71 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:16.076000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 710/1417] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:15 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:16.076987 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b018a89-7daf-49a7-9123-189a2cdffd71 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:16.097964 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b018a89-7daf-49a7-9123-189a2cdffd71 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:16.115748 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97dacc55-34e9-439b-9661-48dce7ad66d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:16.116435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dacc55-34e9-439b-9661-48dce7ad66d3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:16.116744 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dacc55-34e9-439b-9661-48dce7ad66d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:16.117088 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dacc55-34e9-439b-9661-48dce7ad66d3 admin admin] PATH_INFO: `/v3/projects/324b3f5ee1794987b521344fd0960b2d/users/3877a86cf209484ca53112a98c787735/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:16.165491 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abf1b88-03ae-4cd9-b12f-614a465935ba service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:16.169004 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97dacc55-34e9-439b-9661-48dce7ad66d3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=324b3f5ee1794987b521344fd0960b2d, user_id=3877a86cf209484ca53112a98c787735, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:16.175090 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97dacc55-34e9-439b-9661-48dce7ad66d3 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:16.175803 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abf1b88-03ae-4cd9-b12f-614a465935ba service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:16.217470 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 711/1418] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:56:16 2022] PUT /identity/v3/projects/324b3f5ee1794987b521344fd0960b2d/users/3877a86cf209484ca53112a98c787735/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:16.218237 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 708/1419] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:15 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 502 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:16.228372 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35f96d8-8e83-43d0-ba12-b21ffb7678f9 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:16.228647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35f96d8-8e83-43d0-ba12-b21ffb7678f9 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:16.228931 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35f96d8-8e83-43d0-ba12-b21ffb7678f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:16.328760 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-c35f96d8-8e83-43d0-ba12-b21ffb7678f9 None None] MFA Rules not processed for user `3877a86cf209484ca53112a98c787735`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:16.329992 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c35f96d8-8e83-43d0-ba12-b21ffb7678f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:16.408581 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:16.653186 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abf1b88-03ae-4cd9-b12f-614a465935ba service nova] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:16.654888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 712/1420] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:16 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 425 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:16.667068 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abf1b88-03ae-4cd9-b12f-614a465935ba service nova] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:16.867983 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:16.961886 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a665f9-1363-4adf-b745-53cfb3d93025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:16.968497 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a665f9-1363-4adf-b745-53cfb3d93025 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:16.970433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a665f9-1363-4adf-b745-53cfb3d93025 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:16.972186 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aed04be-2184-4c26-aa2e-73276ebcc907 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:16.972373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a665f9-1363-4adf-b745-53cfb3d93025 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:16.972861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aed04be-2184-4c26-aa2e-73276ebcc907 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:16.973335 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aed04be-2184-4c26-aa2e-73276ebcc907 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:16.974641 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a665f9-1363-4adf-b745-53cfb3d93025 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:16.975630 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 713/1421] 10.222.0.16 () {54 vars in 921 bytes} [Sun Oct 30 00:56:16 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:56:16.985553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a665f9-1363-4adf-b745-53cfb3d93025 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:17.067750 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 709/1422] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:56:16 2022] POST /identity/v3/projects => generated 357 bytes in 413 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:56:17.070132 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aed04be-2184-4c26-aa2e-73276ebcc907 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:17.155258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfd6f71-d304-4f35-9b57-4aaa3d7cac04 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:17.156689 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfd6f71-d304-4f35-9b57-4aaa3d7cac04 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:17.157427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfd6f71-d304-4f35-9b57-4aaa3d7cac04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:17.243584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-9cfd6f71-d304-4f35-9b57-4aaa3d7cac04 None None] MFA Rules not processed for user `e7a886715b5b4c1d99f0b37c58cb5417`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:17.246596 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-9cfd6f71-d304-4f35-9b57-4aaa3d7cac04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:17.285766 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aed04be-2184-4c26-aa2e-73276ebcc907 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:17.299924 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f777f8b5-95cc-4f5a-92a9-c4a05ea6f3ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:17.302267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f777f8b5-95cc-4f5a-92a9-c4a05ea6f3ec admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:17.302838 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f777f8b5-95cc-4f5a-92a9-c4a05ea6f3ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:17.303831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f777f8b5-95cc-4f5a-92a9-c4a05ea6f3ec admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:17.304705 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f777f8b5-95cc-4f5a-92a9-c4a05ea6f3ec admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:17.309091 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f777f8b5-95cc-4f5a-92a9-c4a05ea6f3ec admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:17.366123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 714/1423] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:56:17 2022] POST /identity/v3/users => generated 285 bytes in 297 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:56:17.369076 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f777f8b5-95cc-4f5a-92a9-c4a05ea6f3ec admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:17.390794 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f777f8b5-95cc-4f5a-92a9-c4a05ea6f3ec admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:17.408421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47043412-600c-4024-a52b-0f43bca92ebb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:17.409846 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47043412-600c-4024-a52b-0f43bca92ebb admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:17.411103 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47043412-600c-4024-a52b-0f43bca92ebb admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:17.412511 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47043412-600c-4024-a52b-0f43bca92ebb admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:17.413624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47043412-600c-4024-a52b-0f43bca92ebb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:17.416986 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47043412-600c-4024-a52b-0f43bca92ebb admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:17.443633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 715/1424] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:17 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:17.448622 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47043412-600c-4024-a52b-0f43bca92ebb admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:17.450787 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 710/1425] 10.222.0.16 () {58 vars in 1018 bytes} [Sun Oct 30 00:56:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 299 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:17.462682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47043412-600c-4024-a52b-0f43bca92ebb admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:17.472468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc02cd1-40f0-4158-b4d9-64a00b1e934f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:17.474301 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc02cd1-40f0-4158-b4d9-64a00b1e934f admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:17.475193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc02cd1-40f0-4158-b4d9-64a00b1e934f admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:17.476095 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc02cd1-40f0-4158-b4d9-64a00b1e934f admin admin] PATH_INFO: `/v3/projects/450708711051450abc140191202115fd/users/6c0933c2416446e79b47344a7ba65edd/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:17.547157 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc02cd1-40f0-4158-b4d9-64a00b1e934f admin admin] RBAC: Authorizing `identity:create_grant(project_id=450708711051450abc140191202115fd, user_id=6c0933c2416446e79b47344a7ba65edd, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:17.554905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc02cd1-40f0-4158-b4d9-64a00b1e934f admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:17.605887 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 716/1426] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:56:17 2022] PUT /identity/v3/projects/450708711051450abc140191202115fd/users/6c0933c2416446e79b47344a7ba65edd/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:17.609915 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfd6f71-d304-4f35-9b57-4aaa3d7cac04 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:17.632835 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfd6f71-d304-4f35-9b57-4aaa3d7cac04 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:17.788264 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de76933-36a7-43f0-8366-93e1f433ba1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:17.789053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de76933-36a7-43f0-8366-93e1f433ba1f admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:17.789517 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de76933-36a7-43f0-8366-93e1f433ba1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:17.789873 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de76933-36a7-43f0-8366-93e1f433ba1f admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:17.791062 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de76933-36a7-43f0-8366-93e1f433ba1f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:17.793031 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de76933-36a7-43f0-8366-93e1f433ba1f admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:17.848345 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 711/1427] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:17 2022] GET /identity/v3/roles => generated 1420 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:17.852726 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc02cd1-40f0-4158-b4d9-64a00b1e934f admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:17.882925 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc02cd1-40f0-4158-b4d9-64a00b1e934f admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:17.900134 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4344a5c-d44b-4757-94dc-39d444f1acf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:17.902567 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4344a5c-d44b-4757-94dc-39d444f1acf3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:17.904312 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4344a5c-d44b-4757-94dc-39d444f1acf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:17.905724 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4344a5c-d44b-4757-94dc-39d444f1acf3 admin admin] PATH_INFO: `/v3/projects/450708711051450abc140191202115fd/users/6c0933c2416446e79b47344a7ba65edd/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:17.912430 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4344a5c-d44b-4757-94dc-39d444f1acf3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=450708711051450abc140191202115fd, user_id=6c0933c2416446e79b47344a7ba65edd, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:17.928655 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4344a5c-d44b-4757-94dc-39d444f1acf3 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:17.957303 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 717/1428] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:56:17 2022] PUT /identity/v3/projects/450708711051450abc140191202115fd/users/6c0933c2416446e79b47344a7ba65edd/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:17.970880 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a28f140-2940-4f16-b059-9cb492cc69a2 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:17.975122 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a28f140-2940-4f16-b059-9cb492cc69a2 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:17.975580 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a28f140-2940-4f16-b059-9cb492cc69a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:18.064003 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-9a28f140-2940-4f16-b059-9cb492cc69a2 None None] MFA Rules not processed for user `6c0933c2416446e79b47344a7ba65edd`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:18.067661 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-9a28f140-2940-4f16-b059-9cb492cc69a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:18.120749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4344a5c-d44b-4757-94dc-39d444f1acf3 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:18.176446 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:18.243114 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:18.262942 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 712/1429] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:17 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 293 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:18.280360 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a28f140-2940-4f16-b059-9cb492cc69a2 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:18.281837 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-9a28f140-2940-4f16-b059-9cb492cc69a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:18.297598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4344a5c-d44b-4757-94dc-39d444f1acf3 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:18.317954 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd72e98-7949-4037-bc24-49e12df82384 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:18.318781 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd72e98-7949-4037-bc24-49e12df82384 service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:18.319150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd72e98-7949-4037-bc24-49e12df82384 service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:18.319848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd72e98-7949-4037-bc24-49e12df82384 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:18.370757 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd72e98-7949-4037-bc24-49e12df82384 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:18.377325 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd72e98-7949-4037-bc24-49e12df82384 service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:18.410512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 718/1430] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:18.525645 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a28f140-2940-4f16-b059-9cb492cc69a2 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:18.532359 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a61504a-8d0b-4772-b7cd-4a869e2c0e4a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:18.533144 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a61504a-8d0b-4772-b7cd-4a869e2c0e4a service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:18.533535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a61504a-8d0b-4772-b7cd-4a869e2c0e4a service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:18.534397 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a61504a-8d0b-4772-b7cd-4a869e2c0e4a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:18.558278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a61504a-8d0b-4772-b7cd-4a869e2c0e4a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:18.559940 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a61504a-8d0b-4772-b7cd-4a869e2c0e4a service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:18.589874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 713/1431] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:18 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 00:56:18.762384 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd72e98-7949-4037-bc24-49e12df82384 service nova] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:18.775769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5135d6d1-9e36-45a4-b991-33ef7324e87e None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:18.776092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5135d6d1-9e36-45a4-b991-33ef7324e87e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:18.776455 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5135d6d1-9e36-45a4-b991-33ef7324e87e None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:18.778321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 714/1432] 10.222.0.16 () {54 vars in 921 bytes} [Sun Oct 30 00:56:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 00:56:18.785603 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd72e98-7949-4037-bc24-49e12df82384 service nova] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:18.797219 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc9824f-2be0-455e-b9d6-aae69d5d5bec service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:18.797919 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc9824f-2be0-455e-b9d6-aae69d5d5bec service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:18.798353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc9824f-2be0-455e-b9d6-aae69d5d5bec service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:18.798577 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc9824f-2be0-455e-b9d6-aae69d5d5bec service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:18.952521 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc9824f-2be0-455e-b9d6-aae69d5d5bec service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:18.953978 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc9824f-2be0-455e-b9d6-aae69d5d5bec service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:19.019452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 719/1433] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:18 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 255 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:19.021035 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5135d6d1-9e36-45a4-b991-33ef7324e87e None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:19.029321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-5135d6d1-9e36-45a4-b991-33ef7324e87e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:19.029983 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07938c78-c2b0-4b8e-bbe3-de7eeb54b61c None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:19.030267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07938c78-c2b0-4b8e-bbe3-de7eeb54b61c None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:19.030540 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07938c78-c2b0-4b8e-bbe3-de7eeb54b61c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:19.114415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-07938c78-c2b0-4b8e-bbe3-de7eeb54b61c None None] MFA Rules not processed for user `e7a886715b5b4c1d99f0b37c58cb5417`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:19.127271 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-07938c78-c2b0-4b8e-bbe3-de7eeb54b61c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:19.161485 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 720/1434] 10.222.0.16 () {58 vars in 1018 bytes} [Sun Oct 30 00:56:19 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:19.361196 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5135d6d1-9e36-45a4-b991-33ef7324e87e None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:19.381752 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb2e3f7-c44d-4fc8-a136-7143ee41770d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:19.383226 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb2e3f7-c44d-4fc8-a136-7143ee41770d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:19.383688 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb2e3f7-c44d-4fc8-a136-7143ee41770d admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:19.384166 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb2e3f7-c44d-4fc8-a136-7143ee41770d admin admin] PATH_INFO: `/v3/projects/47ea8ebe1bc64f6183b8dca990bb3ae4` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:19.387015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb2e3f7-c44d-4fc8-a136-7143ee41770d admin admin] RBAC: Authorizing `identity:delete_project(project_id=47ea8ebe1bc64f6183b8dca990bb3ae4)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:19.393118 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb2e3f7-c44d-4fc8-a136-7143ee41770d admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:19.545390 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-cfb2e3f7-c44d-4fc8-a136-7143ee41770d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '47ea8ebe1bc64f6183b8dca990bb3ae4', 'request_id': 'req-cfb2e3f7-c44d-4fc8-a136-7143ee41770d'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:19.572934 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-cfb2e3f7-c44d-4fc8-a136-7143ee41770d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '47ea8ebe1bc64f6183b8dca990bb3ae4', 'request_id': 'req-cfb2e3f7-c44d-4fc8-a136-7143ee41770d'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:19.601725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-cfb2e3f7-c44d-4fc8-a136-7143ee41770d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '47ea8ebe1bc64f6183b8dca990bb3ae4', 'request_id': 'req-cfb2e3f7-c44d-4fc8-a136-7143ee41770d'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:19.634446 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-cfb2e3f7-c44d-4fc8-a136-7143ee41770d admin admin] The token cache is being invalidate because project 47ea8ebe1bc64f6183b8dca990bb3ae4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:56:19.635490 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-cfb2e3f7-c44d-4fc8-a136-7143ee41770d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:19.639617 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-cfb2e3f7-c44d-4fc8-a136-7143ee41770d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:19.650582 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 715/1435] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:56:19 2022] DELETE /identity/v3/projects/47ea8ebe1bc64f6183b8dca990bb3ae4 => generated 0 bytes in 625 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:19.651185 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd064d0-9f4d-4330-b2cb-c7a24ef04d4f None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:19.652016 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd064d0-9f4d-4330-b2cb-c7a24ef04d4f None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:19.652481 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd064d0-9f4d-4330-b2cb-c7a24ef04d4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:19.697158 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-1fd064d0-9f4d-4330-b2cb-c7a24ef04d4f None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:19.698543 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-1fd064d0-9f4d-4330-b2cb-c7a24ef04d4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:19.868941 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 721/1436] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:19.869632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd65050f-3d82-45f3-b681-53bc0c3c226a None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:19.870041 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd65050f-3d82-45f3-b681-53bc0c3c226a None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:19.870183 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd65050f-3d82-45f3-b681-53bc0c3c226a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:19.963687 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-bd65050f-3d82-45f3-b681-53bc0c3c226a None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:19.976529 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-bd65050f-3d82-45f3-b681-53bc0c3c226a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:19.981548 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd064d0-9f4d-4330-b2cb-c7a24ef04d4f None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:19.983316 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-1fd064d0-9f4d-4330-b2cb-c7a24ef04d4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:20.020239 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 716/1437] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:56:20.023385 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd65050f-3d82-45f3-b681-53bc0c3c226a None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:20.056638 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd65050f-3d82-45f3-b681-53bc0c3c226a None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:20.066426 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3914db83-682a-47e4-8b3b-906d60661175 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:20.068265 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3914db83-682a-47e4-8b3b-906d60661175 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:20.069016 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3914db83-682a-47e4-8b3b-906d60661175 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:20.070344 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3914db83-682a-47e4-8b3b-906d60661175 admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:20.071735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3914db83-682a-47e4-8b3b-906d60661175 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:56:20.072111 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3914db83-682a-47e4-8b3b-906d60661175 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:20.075198 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3914db83-682a-47e4-8b3b-906d60661175 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:20.081974 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:20.102360 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 717/1438] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:56:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:56:20.103461 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3914db83-682a-47e4-8b3b-906d60661175 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:20.142224 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3914db83-682a-47e4-8b3b-906d60661175 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:20.174416 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669e1706-50d5-4e42-b64b-eaf53cd9701e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:20.174416 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669e1706-50d5-4e42-b64b-eaf53cd9701e admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:20.174416 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669e1706-50d5-4e42-b64b-eaf53cd9701e admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:20.174416 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669e1706-50d5-4e42-b64b-eaf53cd9701e admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:20.174900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669e1706-50d5-4e42-b64b-eaf53cd9701e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:20.174900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669e1706-50d5-4e42-b64b-eaf53cd9701e admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:20.241144 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd064d0-9f4d-4330-b2cb-c7a24ef04d4f None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:20.244749 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 718/1439] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:56:20 2022] POST /identity/v3/projects => generated 373 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:56:20.249735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669e1706-50d5-4e42-b64b-eaf53cd9701e admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:20.255409 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:20.426089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0443abd7-8169-434a-95c6-089e4abf1ef7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:20.426724 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0443abd7-8169-434a-95c6-089e4abf1ef7 service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:20.427001 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0443abd7-8169-434a-95c6-089e4abf1ef7 service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:20.428305 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0443abd7-8169-434a-95c6-089e4abf1ef7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:20.430398 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-0443abd7-8169-434a-95c6-089e4abf1ef7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:20.432238 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669e1706-50d5-4e42-b64b-eaf53cd9701e admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:20.448538 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0443abd7-8169-434a-95c6-089e4abf1ef7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:20.450510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0443abd7-8169-434a-95c6-089e4abf1ef7 service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:20.452667 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd8cd6c-fca8-4060-b409-a5535196dedb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:20.453277 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd8cd6c-fca8-4060-b409-a5535196dedb admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:20.453549 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd8cd6c-fca8-4060-b409-a5535196dedb admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:20.453816 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd8cd6c-fca8-4060-b409-a5535196dedb admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:20.454491 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd8cd6c-fca8-4060-b409-a5535196dedb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:20.461082 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd8cd6c-fca8-4060-b409-a5535196dedb admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:20.487921 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 722/1440] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 511 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:20.515909 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 719/1441] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:56:20 2022] POST /identity/v3/users => generated 287 bytes in 267 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:56:20.519012 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0443abd7-8169-434a-95c6-089e4abf1ef7 service nova] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:20.534662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0443abd7-8169-434a-95c6-089e4abf1ef7 service nova] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:20.550463 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b820380-b4e2-4e75-a308-e36000ead7ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:20.552137 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b820380-b4e2-4e75-a308-e36000ead7ca admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:20.552775 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b820380-b4e2-4e75-a308-e36000ead7ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:20.554044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b820380-b4e2-4e75-a308-e36000ead7ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:20.556017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b820380-b4e2-4e75-a308-e36000ead7ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:20.560330 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b820380-b4e2-4e75-a308-e36000ead7ca admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:20.593101 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 723/1442] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:20 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:20.596904 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd8cd6c-fca8-4060-b409-a5535196dedb admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:20.611269 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd8cd6c-fca8-4060-b409-a5535196dedb admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:20.618928 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c8b96e-005b-4364-85a3-4247a2390003 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:20.619632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c8b96e-005b-4364-85a3-4247a2390003 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:20.620017 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c8b96e-005b-4364-85a3-4247a2390003 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:20.620347 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c8b96e-005b-4364-85a3-4247a2390003 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:20.621200 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c8b96e-005b-4364-85a3-4247a2390003 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:20.623552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c8b96e-005b-4364-85a3-4247a2390003 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:20.639782 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 720/1443] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:20 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:20.643639 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b820380-b4e2-4e75-a308-e36000ead7ca admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:20.660058 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b820380-b4e2-4e75-a308-e36000ead7ca admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:20.673668 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8848b5-eaca-4ba7-af83-43325898bd2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:20.676412 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8848b5-eaca-4ba7-af83-43325898bd2a admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:20.676772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8848b5-eaca-4ba7-af83-43325898bd2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:20.677418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8848b5-eaca-4ba7-af83-43325898bd2a admin admin] PATH_INFO: `/v3/projects/e9c26f5f988f415693f0d561736976d2/users/cebabbfa35d84f28b02f3d46797a3129/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:20.724984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8848b5-eaca-4ba7-af83-43325898bd2a admin admin] RBAC: Authorizing `identity:create_grant(project_id=e9c26f5f988f415693f0d561736976d2, user_id=cebabbfa35d84f28b02f3d46797a3129, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:20.730743 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8848b5-eaca-4ba7-af83-43325898bd2a admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:20.756826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 724/1444] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:56:20 2022] PUT /identity/v3/projects/e9c26f5f988f415693f0d561736976d2/users/cebabbfa35d84f28b02f3d46797a3129/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:20.786561 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4d893a-9838-449e-be2e-8eacbac8999e None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:20.787186 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4d893a-9838-449e-be2e-8eacbac8999e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:20.788033 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4d893a-9838-449e-be2e-8eacbac8999e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:20.923857 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-8a4d893a-9838-449e-be2e-8eacbac8999e None None] MFA Rules not processed for user `cebabbfa35d84f28b02f3d46797a3129`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:20.926683 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-8a4d893a-9838-449e-be2e-8eacbac8999e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:21.218310 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8848b5-eaca-4ba7-af83-43325898bd2a admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:21.220803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ab8848b5-eaca-4ba7-af83-43325898bd2a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:21.250252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 721/1445] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:20 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 466 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:21.259240 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4d893a-9838-449e-be2e-8eacbac8999e None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:21.407993 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:21.416350 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8848b5-eaca-4ba7-af83-43325898bd2a admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:21.428780 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a595fc6a-497f-4dfa-8a21-6601a1cf16ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:21.429846 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a595fc6a-497f-4dfa-8a21-6601a1cf16ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:21.430357 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a595fc6a-497f-4dfa-8a21-6601a1cf16ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:21.431692 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a595fc6a-497f-4dfa-8a21-6601a1cf16ca admin admin] PATH_INFO: `/v3/users/391b5a6cb3bd4569ad3df5663194d1f4` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:21.434148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a595fc6a-497f-4dfa-8a21-6601a1cf16ca admin admin] RBAC: Authorizing `identity:delete_user(user_id=391b5a6cb3bd4569ad3df5663194d1f4)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:21.436108 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a595fc6a-497f-4dfa-8a21-6601a1cf16ca admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:21.442450 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4d893a-9838-449e-be2e-8eacbac8999e None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:21.447819 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9750d156-3d91-49de-a2af-069cc0836325 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:21.448491 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9750d156-3d91-49de-a2af-069cc0836325 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:21.448668 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9750d156-3d91-49de-a2af-069cc0836325 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:21.448966 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9750d156-3d91-49de-a2af-069cc0836325 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:21.449546 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9750d156-3d91-49de-a2af-069cc0836325 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:21.452117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9750d156-3d91-49de-a2af-069cc0836325 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:21.497607 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 722/1446] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:56:21 2022] POST /identity/v3/projects => generated 373 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:56:21.499739 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9750d156-3d91-49de-a2af-069cc0836325 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:21.563823 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a595fc6a-497f-4dfa-8a21-6601a1cf16ca admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '391b5a6cb3bd4569ad3df5663194d1f4'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:21.576257 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a595fc6a-497f-4dfa-8a21-6601a1cf16ca admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '391b5a6cb3bd4569ad3df5663194d1f4'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:21.578147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a595fc6a-497f-4dfa-8a21-6601a1cf16ca admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '391b5a6cb3bd4569ad3df5663194d1f4'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:21.589596 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9750d156-3d91-49de-a2af-069cc0836325 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:21.594799 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e918bb3-b9ae-4e62-8533-a778fc009904 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:21.595485 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e918bb3-b9ae-4e62-8533-a778fc009904 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:21.595882 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e918bb3-b9ae-4e62-8533-a778fc009904 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:21.596263 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e918bb3-b9ae-4e62-8533-a778fc009904 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:21.596961 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e918bb3-b9ae-4e62-8533-a778fc009904 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:21.599241 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e918bb3-b9ae-4e62-8533-a778fc009904 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:21.602090 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:INSERT=2,keystone:DELETE=9 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:21.631268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a595fc6a-497f-4dfa-8a21-6601a1cf16ca admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '391b5a6cb3bd4569ad3df5663194d1f4'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:21.637088 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 723/1447] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:56:21 2022] POST /identity/v3/users => generated 293 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:56:21.641077 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e918bb3-b9ae-4e62-8533-a778fc009904 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:21.642944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a595fc6a-497f-4dfa-8a21-6601a1cf16ca admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '391b5a6cb3bd4569ad3df5663194d1f4'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:21.649706 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-8e918bb3-b9ae-4e62-8533-a778fc009904 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:21.665624 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e918bb3-b9ae-4e62-8533-a778fc009904 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:21.667219 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 725/1448] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:56:21 2022] DELETE /identity/v3/users/391b5a6cb3bd4569ad3df5663194d1f4 => generated 0 bytes in 449 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:21.671862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f660bc-6a67-4053-999a-70844775037f None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:21.672141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f660bc-6a67-4053-999a-70844775037f None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:21.672406 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f660bc-6a67-4053-999a-70844775037f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:21.721603 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-88f660bc-6a67-4053-999a-70844775037f None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:21.726874 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-88f660bc-6a67-4053-999a-70844775037f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:21.826785 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a82c11-62b0-4e58-b9cc-9bb76e7c48a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:21.827769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a82c11-62b0-4e58-b9cc-9bb76e7c48a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:21.831223 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a82c11-62b0-4e58-b9cc-9bb76e7c48a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:21.831813 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a82c11-62b0-4e58-b9cc-9bb76e7c48a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:21.832790 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a82c11-62b0-4e58-b9cc-9bb76e7c48a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:21.838630 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a82c11-62b0-4e58-b9cc-9bb76e7c48a9 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:21.864826 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 724/1449] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:21 2022] GET /identity/v3/roles => generated 1420 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:21.872529 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a82c11-62b0-4e58-b9cc-9bb76e7c48a9 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:21.896564 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a82c11-62b0-4e58-b9cc-9bb76e7c48a9 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:21.898944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 726/1450] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:56:21.910553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8324376d-4e75-4638-9dee-1d35df8c60bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:21.910789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8324376d-4e75-4638-9dee-1d35df8c60bd admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:21.911680 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8324376d-4e75-4638-9dee-1d35df8c60bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:21.911680 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8324376d-4e75-4638-9dee-1d35df8c60bd admin admin] PATH_INFO: `/v3/projects/ea216333fa5c45c79f656ee70fad8b8b/users/e0bc45577a024bcdb4fd97faa2c988ad/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:21.949233 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8324376d-4e75-4638-9dee-1d35df8c60bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea216333fa5c45c79f656ee70fad8b8b, user_id=e0bc45577a024bcdb4fd97faa2c988ad, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:21.952185 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8324376d-4e75-4638-9dee-1d35df8c60bd admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:21.959540 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f660bc-6a67-4053-999a-70844775037f None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:21.961788 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-88f660bc-6a67-4053-999a-70844775037f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:21.980542 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 725/1451] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:56:21 2022] PUT /identity/v3/projects/ea216333fa5c45c79f656ee70fad8b8b/users/e0bc45577a024bcdb4fd97faa2c988ad/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:21.982436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8324376d-4e75-4638-9dee-1d35df8c60bd admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:22.105912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f660bc-6a67-4053-999a-70844775037f None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:22.155108 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:22.236630 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bffd9fa1-cab3-4e65-9c95-0975612a1d16 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:22.237521 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffd9fa1-cab3-4e65-9c95-0975612a1d16 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:22.237890 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffd9fa1-cab3-4e65-9c95-0975612a1d16 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:22.238249 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffd9fa1-cab3-4e65-9c95-0975612a1d16 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:22.281072 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8324376d-4e75-4638-9dee-1d35df8c60bd admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:22.296653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0937d9db-0f96-46d5-9a17-f9b492e56f51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:22.298701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0937d9db-0f96-46d5-9a17-f9b492e56f51 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:22.299634 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0937d9db-0f96-46d5-9a17-f9b492e56f51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:22.302284 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0937d9db-0f96-46d5-9a17-f9b492e56f51 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:22.306855 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0937d9db-0f96-46d5-9a17-f9b492e56f51 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:22.312787 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0937d9db-0f96-46d5-9a17-f9b492e56f51 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:22.355029 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 726/1452] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:21 2022] GET /identity/v3/roles => generated 1420 bytes in 367 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:22.358084 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0937d9db-0f96-46d5-9a17-f9b492e56f51 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:22.369172 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:22.392413 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0937d9db-0f96-46d5-9a17-f9b492e56f51 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:22.392708 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bffd9fa1-cab3-4e65-9c95-0975612a1d16 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:22.394185 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bffd9fa1-cab3-4e65-9c95-0975612a1d16 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:22.406584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cdd974-1d82-408e-9635-62a07c1759e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:22.408592 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cdd974-1d82-408e-9635-62a07c1759e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:22.411299 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cdd974-1d82-408e-9635-62a07c1759e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:22.411590 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cdd974-1d82-408e-9635-62a07c1759e7 admin admin] PATH_INFO: `/v3/projects/ea216333fa5c45c79f656ee70fad8b8b/users/e0bc45577a024bcdb4fd97faa2c988ad/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:22.418453 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cdd974-1d82-408e-9635-62a07c1759e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea216333fa5c45c79f656ee70fad8b8b, user_id=e0bc45577a024bcdb4fd97faa2c988ad, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:22.420064 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 727/1453] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 461 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:22.426904 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cdd974-1d82-408e-9635-62a07c1759e7 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:22.480520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 727/1454] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:56:22 2022] PUT /identity/v3/projects/ea216333fa5c45c79f656ee70fad8b8b/users/e0bc45577a024bcdb4fd97faa2c988ad/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:22.491510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34881258-e93d-41c4-8296-1d9b038a8c79 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:22.491848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34881258-e93d-41c4-8296-1d9b038a8c79 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:22.492128 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34881258-e93d-41c4-8296-1d9b038a8c79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:22.567211 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-34881258-e93d-41c4-8296-1d9b038a8c79 None None] MFA Rules not processed for user `e0bc45577a024bcdb4fd97faa2c988ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:22.570470 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-34881258-e93d-41c4-8296-1d9b038a8c79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:22.786295 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cdd974-1d82-408e-9635-62a07c1759e7 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:22.786628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 728/1455] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:22 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 296 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:22.790522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-c1cdd974-1d82-408e-9635-62a07c1759e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:22.905331 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cdd974-1d82-408e-9635-62a07c1759e7 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:22.910987 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80d6e1f-23f0-4498-984a-8e401ba2ba39 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:22.913069 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80d6e1f-23f0-4498-984a-8e401ba2ba39 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:22.913512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80d6e1f-23f0-4498-984a-8e401ba2ba39 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:22.913883 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80d6e1f-23f0-4498-984a-8e401ba2ba39 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:22.964654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80d6e1f-23f0-4498-984a-8e401ba2ba39 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:22.983980 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80d6e1f-23f0-4498-984a-8e401ba2ba39 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:23.030835 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 728/1456] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:22 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:23.227878 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34881258-e93d-41c4-8296-1d9b038a8c79 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:23.241176 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34881258-e93d-41c4-8296-1d9b038a8c79 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:23.251259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03405092-7f0f-49e3-a127-d14b2472b217 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:23.251615 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03405092-7f0f-49e3-a127-d14b2472b217 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:23.253185 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03405092-7f0f-49e3-a127-d14b2472b217 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:23.253470 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03405092-7f0f-49e3-a127-d14b2472b217 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:23.258452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-03405092-7f0f-49e3-a127-d14b2472b217 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:23.395472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03405092-7f0f-49e3-a127-d14b2472b217 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:23.398894 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03405092-7f0f-49e3-a127-d14b2472b217 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:23.426183 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 729/1457] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:23 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:24.036834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80d6e1f-23f0-4498-984a-8e401ba2ba39 service cinder] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:24.038377 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-f80d6e1f-23f0-4498-984a-8e401ba2ba39 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:24.142650 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80d6e1f-23f0-4498-984a-8e401ba2ba39 service cinder] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:24.149770 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cf13f0-18e7-4bbb-9baf-82184fd6c9ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:24.151224 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cf13f0-18e7-4bbb-9baf-82184fd6c9ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:24.151504 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cf13f0-18e7-4bbb-9baf-82184fd6c9ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:24.151822 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cf13f0-18e7-4bbb-9baf-82184fd6c9ad admin admin] PATH_INFO: `/v3/projects/782a4eade27d42298bbf4519f07d3e7f` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:24.153178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cf13f0-18e7-4bbb-9baf-82184fd6c9ad admin admin] RBAC: Authorizing `identity:delete_project(project_id=782a4eade27d42298bbf4519f07d3e7f)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:24.155753 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cf13f0-18e7-4bbb-9baf-82184fd6c9ad admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:24.222925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-71cf13f0-18e7-4bbb-9baf-82184fd6c9ad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '782a4eade27d42298bbf4519f07d3e7f', 'request_id': 'req-71cf13f0-18e7-4bbb-9baf-82184fd6c9ad'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:24.228589 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=93,keystone:INSERT=1,keystone:DELETE=6 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:24.230240 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-71cf13f0-18e7-4bbb-9baf-82184fd6c9ad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '782a4eade27d42298bbf4519f07d3e7f', 'request_id': 'req-71cf13f0-18e7-4bbb-9baf-82184fd6c9ad'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:24.241847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-71cf13f0-18e7-4bbb-9baf-82184fd6c9ad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '782a4eade27d42298bbf4519f07d3e7f', 'request_id': 'req-71cf13f0-18e7-4bbb-9baf-82184fd6c9ad'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:24.260009 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-71cf13f0-18e7-4bbb-9baf-82184fd6c9ad admin admin] The token cache is being invalidate because project 782a4eade27d42298bbf4519f07d3e7f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:56:24.260317 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-71cf13f0-18e7-4bbb-9baf-82184fd6c9ad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:24.261189 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-71cf13f0-18e7-4bbb-9baf-82184fd6c9ad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:24.263128 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 729/1458] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:56:24 2022] DELETE /identity/v3/projects/782a4eade27d42298bbf4519f07d3e7f => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:24.267956 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51237bc-84b1-4d11-b226-04cc4be39fb9 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:24.268426 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51237bc-84b1-4d11-b226-04cc4be39fb9 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:24.268503 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51237bc-84b1-4d11-b226-04cc4be39fb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:24.295576 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-a51237bc-84b1-4d11-b226-04cc4be39fb9 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:24.296959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-a51237bc-84b1-4d11-b226-04cc4be39fb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:24.364130 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 730/1459] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:24.372457 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3ec4bc-74a2-4e2b-9c49-0aeac6277214 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:24.372800 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3ec4bc-74a2-4e2b-9c49-0aeac6277214 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:24.373127 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3ec4bc-74a2-4e2b-9c49-0aeac6277214 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:24.392900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-be3ec4bc-74a2-4e2b-9c49-0aeac6277214 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:24.394158 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-be3ec4bc-74a2-4e2b-9c49-0aeac6277214 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:24.402787 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 730/1460] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:24.407030 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51237bc-84b1-4d11-b226-04cc4be39fb9 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:24.416798 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51237bc-84b1-4d11-b226-04cc4be39fb9 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:24.421408 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d094011-c005-4bb2-a9e2-3842314f961e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:24.422126 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d094011-c005-4bb2-a9e2-3842314f961e admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:24.422509 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d094011-c005-4bb2-a9e2-3842314f961e admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:24.422866 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d094011-c005-4bb2-a9e2-3842314f961e admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:24.423622 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d094011-c005-4bb2-a9e2-3842314f961e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:56:24.424155 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d094011-c005-4bb2-a9e2-3842314f961e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:24.425910 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d094011-c005-4bb2-a9e2-3842314f961e admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:24.434690 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:24.438152 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 731/1461] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:56:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:56:24.442483 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3ec4bc-74a2-4e2b-9c49-0aeac6277214 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:24.452363 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3ec4bc-74a2-4e2b-9c49-0aeac6277214 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:24.456881 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ee6491-40da-4ec0-9129-9ee3dea76d51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:24.457487 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ee6491-40da-4ec0-9129-9ee3dea76d51 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:24.457763 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ee6491-40da-4ec0-9129-9ee3dea76d51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:24.458013 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ee6491-40da-4ec0-9129-9ee3dea76d51 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:24.458622 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ee6491-40da-4ec0-9129-9ee3dea76d51 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:24.460909 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ee6491-40da-4ec0-9129-9ee3dea76d51 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:24.488938 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 731/1462] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:56:24 2022] POST /identity/v3/projects => generated 365 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:56:24.493897 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d094011-c005-4bb2-a9e2-3842314f961e admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:24.508158 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d094011-c005-4bb2-a9e2-3842314f961e admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:24.574535 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfc1930-3b4f-4853-8429-be1198c7e9d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:24.575229 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfc1930-3b4f-4853-8429-be1198c7e9d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:24.575580 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfc1930-3b4f-4853-8429-be1198c7e9d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:24.576046 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfc1930-3b4f-4853-8429-be1198c7e9d8 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:24.576749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbfc1930-3b4f-4853-8429-be1198c7e9d8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:24.578775 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbfc1930-3b4f-4853-8429-be1198c7e9d8 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:24.608566 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 732/1463] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:56:24 2022] POST /identity/v3/users => generated 283 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:56:24.609975 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ee6491-40da-4ec0-9129-9ee3dea76d51 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:24.620661 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ee6491-40da-4ec0-9129-9ee3dea76d51 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:24.625176 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b798ca47-e556-41ba-8db9-7790257afc93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:24.625782 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b798ca47-e556-41ba-8db9-7790257afc93 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:24.626056 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b798ca47-e556-41ba-8db9-7790257afc93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:24.626326 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b798ca47-e556-41ba-8db9-7790257afc93 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:24.627414 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b798ca47-e556-41ba-8db9-7790257afc93 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:24.628646 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b798ca47-e556-41ba-8db9-7790257afc93 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:24.641832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 732/1464] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:24 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:24.643354 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfc1930-3b4f-4853-8429-be1198c7e9d8 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:24.653958 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfc1930-3b4f-4853-8429-be1198c7e9d8 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:24.658963 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231eee3d-7e26-441d-97da-caf9ce45373e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:24.659571 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231eee3d-7e26-441d-97da-caf9ce45373e admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:24.659882 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231eee3d-7e26-441d-97da-caf9ce45373e admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:24.660180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231eee3d-7e26-441d-97da-caf9ce45373e admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:24.660896 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-231eee3d-7e26-441d-97da-caf9ce45373e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:24.662766 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-231eee3d-7e26-441d-97da-caf9ce45373e admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:24.674191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 733/1465] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:24 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:24.678420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b798ca47-e556-41ba-8db9-7790257afc93 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:24.690511 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b798ca47-e556-41ba-8db9-7790257afc93 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:24.695783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c670fec-7f6c-41cf-8cc2-e50c8119aef7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:24.696389 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c670fec-7f6c-41cf-8cc2-e50c8119aef7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:24.696659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c670fec-7f6c-41cf-8cc2-e50c8119aef7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:24.696928 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c670fec-7f6c-41cf-8cc2-e50c8119aef7 admin admin] PATH_INFO: `/v3/projects/bfc270e7b5fe461f985c84fc087b54bc/users/0dc87171c5c44091b887b8f719a01ca3/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:24.714129 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c670fec-7f6c-41cf-8cc2-e50c8119aef7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bfc270e7b5fe461f985c84fc087b54bc, user_id=0dc87171c5c44091b887b8f719a01ca3, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:24.716832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c670fec-7f6c-41cf-8cc2-e50c8119aef7 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:24.732061 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 733/1466] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:56:24 2022] PUT /identity/v3/projects/bfc270e7b5fe461f985c84fc087b54bc/users/0dc87171c5c44091b887b8f719a01ca3/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:24.746967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f136a7b5-10ec-4dc5-b387-8cc473362595 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:24.746967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f136a7b5-10ec-4dc5-b387-8cc473362595 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:24.746967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f136a7b5-10ec-4dc5-b387-8cc473362595 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:24.778669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-f136a7b5-10ec-4dc5-b387-8cc473362595 None None] MFA Rules not processed for user `0dc87171c5c44091b887b8f719a01ca3`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:24.780040 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-f136a7b5-10ec-4dc5-b387-8cc473362595 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:24.859995 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 734/1467] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:24 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:24.864716 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c670fec-7f6c-41cf-8cc2-e50c8119aef7 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:24.950369 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c670fec-7f6c-41cf-8cc2-e50c8119aef7 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:24.956886 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b253bf-c2dc-43a4-9fea-dc59d63f9599 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:24.957620 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b253bf-c2dc-43a4-9fea-dc59d63f9599 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:24.957903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b253bf-c2dc-43a4-9fea-dc59d63f9599 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:24.958173 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b253bf-c2dc-43a4-9fea-dc59d63f9599 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:24.958988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b253bf-c2dc-43a4-9fea-dc59d63f9599 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:24.961313 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b253bf-c2dc-43a4-9fea-dc59d63f9599 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:24.996338 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 734/1468] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:56:24 2022] POST /identity/v3/projects => generated 363 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:56:25.001180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f136a7b5-10ec-4dc5-b387-8cc473362595 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:25.013087 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f136a7b5-10ec-4dc5-b387-8cc473362595 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:25.075031 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fcde8f-6a75-4be9-827d-037e6cd68147 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:25.075561 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fcde8f-6a75-4be9-827d-037e6cd68147 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:25.075874 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fcde8f-6a75-4be9-827d-037e6cd68147 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:25.076136 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fcde8f-6a75-4be9-827d-037e6cd68147 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:25.076762 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0fcde8f-6a75-4be9-827d-037e6cd68147 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:25.078854 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0fcde8f-6a75-4be9-827d-037e6cd68147 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:25.095062 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:25.105912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 735/1469] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:56:24 2022] POST /identity/v3/users => generated 288 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:56:25.108208 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b253bf-c2dc-43a4-9fea-dc59d63f9599 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:25.119402 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b253bf-c2dc-43a4-9fea-dc59d63f9599 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:25.124648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce898964-281f-4e7e-9844-bcd1bc782765 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:25.125444 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce898964-281f-4e7e-9844-bcd1bc782765 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:25.125694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce898964-281f-4e7e-9844-bcd1bc782765 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:25.126006 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce898964-281f-4e7e-9844-bcd1bc782765 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:25.126637 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce898964-281f-4e7e-9844-bcd1bc782765 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:25.128388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce898964-281f-4e7e-9844-bcd1bc782765 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:25.139957 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 735/1470] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:25 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:25.144346 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fcde8f-6a75-4be9-827d-037e6cd68147 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:25.154829 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fcde8f-6a75-4be9-827d-037e6cd68147 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:25.160976 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82469b75-0cd1-4919-97a8-cb230f71e07a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:25.161685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82469b75-0cd1-4919-97a8-cb230f71e07a admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:25.162011 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82469b75-0cd1-4919-97a8-cb230f71e07a admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:25.162415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82469b75-0cd1-4919-97a8-cb230f71e07a admin admin] PATH_INFO: `/v3/projects/6db9548154184b07a1186d248538f2c0/users/768033c20cb643699b7824b9b619e621/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:25.198390 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82469b75-0cd1-4919-97a8-cb230f71e07a admin admin] RBAC: Authorizing `identity:create_grant(project_id=6db9548154184b07a1186d248538f2c0, user_id=768033c20cb643699b7824b9b619e621, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:25.201684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82469b75-0cd1-4919-97a8-cb230f71e07a admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:25.224316 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 736/1471] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:56:25 2022] PUT /identity/v3/projects/6db9548154184b07a1186d248538f2c0/users/768033c20cb643699b7824b9b619e621/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:25.226752 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce898964-281f-4e7e-9844-bcd1bc782765 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:25.243968 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce898964-281f-4e7e-9844-bcd1bc782765 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:25.339947 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:25.356448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d7a56b-7b3f-4bf7-8cb7-14a54fa6c1fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:25.357419 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d7a56b-7b3f-4bf7-8cb7-14a54fa6c1fb admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:25.358150 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d7a56b-7b3f-4bf7-8cb7-14a54fa6c1fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:25.358746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d7a56b-7b3f-4bf7-8cb7-14a54fa6c1fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:25.359949 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d7a56b-7b3f-4bf7-8cb7-14a54fa6c1fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:25.362185 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d7a56b-7b3f-4bf7-8cb7-14a54fa6c1fb admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:25.409142 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 736/1472] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:25 2022] GET /identity/v3/roles => generated 1420 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:25.413390 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82469b75-0cd1-4919-97a8-cb230f71e07a admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:25.426074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82469b75-0cd1-4919-97a8-cb230f71e07a admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:25.439642 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f542013-69ef-4104-81a4-8d894831ff66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:25.440701 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f542013-69ef-4104-81a4-8d894831ff66 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:25.441035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f542013-69ef-4104-81a4-8d894831ff66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:25.441376 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f542013-69ef-4104-81a4-8d894831ff66 admin admin] PATH_INFO: `/v3/projects/6db9548154184b07a1186d248538f2c0/users/768033c20cb643699b7824b9b619e621/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:25.443425 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f542013-69ef-4104-81a4-8d894831ff66 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6db9548154184b07a1186d248538f2c0, user_id=768033c20cb643699b7824b9b619e621, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:25.446864 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f542013-69ef-4104-81a4-8d894831ff66 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:25.475635 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 737/1473] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:56:25 2022] PUT /identity/v3/projects/6db9548154184b07a1186d248538f2c0/users/768033c20cb643699b7824b9b619e621/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 00:56:25.476363 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ef964c-b8d9-4d1b-b14c-4c943f6a8979 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:25.476853 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ef964c-b8d9-4d1b-b14c-4c943f6a8979 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:25.477065 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ef964c-b8d9-4d1b-b14c-4c943f6a8979 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:25.527355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-22ef964c-b8d9-4d1b-b14c-4c943f6a8979 None None] MFA Rules not processed for user `768033c20cb643699b7824b9b619e621`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:25.528872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-22ef964c-b8d9-4d1b-b14c-4c943f6a8979 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:25.638355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 737/1474] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:25 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:56:25.654405 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f542013-69ef-4104-81a4-8d894831ff66 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:25.656258 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-9f542013-69ef-4104-81a4-8d894831ff66 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:25.715820 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f542013-69ef-4104-81a4-8d894831ff66 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:25.719473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64920486-6e02-4474-89e7-3f16a61b198a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:25.720119 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64920486-6e02-4474-89e7-3f16a61b198a service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:25.720398 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64920486-6e02-4474-89e7-3f16a61b198a service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:25.720665 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64920486-6e02-4474-89e7-3f16a61b198a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:25.730117 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64920486-6e02-4474-89e7-3f16a61b198a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:25.731581 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64920486-6e02-4474-89e7-3f16a61b198a service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:25.741105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 738/1475] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:25 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:25.831467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ef964c-b8d9-4d1b-b14c-4c943f6a8979 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:25.850011 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ef964c-b8d9-4d1b-b14c-4c943f6a8979 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:25.855111 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2441f83-4628-4232-9cb8-d8a77eb29838 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:25.858663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2441f83-4628-4232-9cb8-d8a77eb29838 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:25.859053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2441f83-4628-4232-9cb8-d8a77eb29838 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:25.859811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2441f83-4628-4232-9cb8-d8a77eb29838 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:25.939399 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2441f83-4628-4232-9cb8-d8a77eb29838 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:25.941057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2441f83-4628-4232-9cb8-d8a77eb29838 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:25.962134 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 738/1476] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:25 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:35.736801 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=42,keystone:INSERT=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:35.947310 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=66 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:43.894234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3964c459-d2c1-4330-a558-11c9496337f2 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:43.894773 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3964c459-d2c1-4330-a558-11c9496337f2 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:43.894773 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3964c459-d2c1-4330-a558-11c9496337f2 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:43.896448 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 739/1477] 10.222.0.16 () {54 vars in 921 bytes} [Sun Oct 30 00:56:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:56:43.910793 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d66f34-d9e8-4555-a00c-ae7a52cb05cd None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:43.911270 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d66f34-d9e8-4555-a00c-ae7a52cb05cd None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:43.911382 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d66f34-d9e8-4555-a00c-ae7a52cb05cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:43.934137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-25d66f34-d9e8-4555-a00c-ae7a52cb05cd None None] MFA Rules not processed for user `e7a886715b5b4c1d99f0b37c58cb5417`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:43.935343 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-25d66f34-d9e8-4555-a00c-ae7a52cb05cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:44.015418 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 739/1478] 10.222.0.16 () {58 vars in 1018 bytes} [Sun Oct 30 00:56:43 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:56:44.028154 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3964c459-d2c1-4330-a558-11c9496337f2 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:44.030024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3964c459-d2c1-4330-a558-11c9496337f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:44.047923 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3964c459-d2c1-4330-a558-11c9496337f2 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:44.054825 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9ce2e6-cb6d-4b78-927a-2a7385e6acaf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:44.055627 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9ce2e6-cb6d-4b78-927a-2a7385e6acaf service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:44.055937 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9ce2e6-cb6d-4b78-927a-2a7385e6acaf service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:44.056199 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9ce2e6-cb6d-4b78-927a-2a7385e6acaf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:44.065679 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9ce2e6-cb6d-4b78-927a-2a7385e6acaf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:44.067317 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9ce2e6-cb6d-4b78-927a-2a7385e6acaf service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:44.078483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 740/1479] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:44 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 00:56:47.422221 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d66f34-d9e8-4555-a00c-ae7a52cb05cd None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:47.492993 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d66f34-d9e8-4555-a00c-ae7a52cb05cd None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:47.497729 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d280b80e-c0fd-465d-8bfd-be2abef8b3df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:47.498413 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d280b80e-c0fd-465d-8bfd-be2abef8b3df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:47.498794 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d280b80e-c0fd-465d-8bfd-be2abef8b3df admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:47.499102 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d280b80e-c0fd-465d-8bfd-be2abef8b3df admin admin] PATH_INFO: `/v3/users/0dc87171c5c44091b887b8f719a01ca3` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:47.500255 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d280b80e-c0fd-465d-8bfd-be2abef8b3df admin admin] RBAC: Authorizing `identity:delete_user(user_id=0dc87171c5c44091b887b8f719a01ca3)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:47.502489 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d280b80e-c0fd-465d-8bfd-be2abef8b3df admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:47.565269 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-d280b80e-c0fd-465d-8bfd-be2abef8b3df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0dc87171c5c44091b887b8f719a01ca3'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:47.579789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-d280b80e-c0fd-465d-8bfd-be2abef8b3df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0dc87171c5c44091b887b8f719a01ca3'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:47.580449 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-d280b80e-c0fd-465d-8bfd-be2abef8b3df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0dc87171c5c44091b887b8f719a01ca3'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:47.589019 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-d280b80e-c0fd-465d-8bfd-be2abef8b3df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0dc87171c5c44091b887b8f719a01ca3'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:47.600845 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-d280b80e-c0fd-465d-8bfd-be2abef8b3df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0dc87171c5c44091b887b8f719a01ca3'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:47.606743 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 740/1480] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:56:47 2022] DELETE /identity/v3/users/0dc87171c5c44091b887b8f719a01ca3 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:47.611594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b83e0e0-899a-47a0-898c-2f60f80739b2 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:47.612089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b83e0e0-899a-47a0-898c-2f60f80739b2 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:47.612089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b83e0e0-899a-47a0-898c-2f60f80739b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:47.632321 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-6b83e0e0-899a-47a0-898c-2f60f80739b2 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:47.633739 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-6b83e0e0-899a-47a0-898c-2f60f80739b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:47.752484 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 741/1481] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:47.777334 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d280b80e-c0fd-465d-8bfd-be2abef8b3df admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:47.779090 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d280b80e-c0fd-465d-8bfd-be2abef8b3df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:47.805813 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:47.889742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d280b80e-c0fd-465d-8bfd-be2abef8b3df admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:47.898208 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c471c144-a0b7-48e4-ac03-d097ca081242 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:47.898911 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c471c144-a0b7-48e4-ac03-d097ca081242 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:47.899188 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c471c144-a0b7-48e4-ac03-d097ca081242 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:47.899518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c471c144-a0b7-48e4-ac03-d097ca081242 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:47.910362 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c471c144-a0b7-48e4-ac03-d097ca081242 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:47.911850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c471c144-a0b7-48e4-ac03-d097ca081242 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:47.924157 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 741/1482] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:48.393323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b83e0e0-899a-47a0-898c-2f60f80739b2 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:48.394807 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-6b83e0e0-899a-47a0-898c-2f60f80739b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:48.408626 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b83e0e0-899a-47a0-898c-2f60f80739b2 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:48.414495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965cd10d-a004-42ab-afac-5b2f584ec510 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:48.415147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965cd10d-a004-42ab-afac-5b2f584ec510 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:48.415419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965cd10d-a004-42ab-afac-5b2f584ec510 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:48.416078 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965cd10d-a004-42ab-afac-5b2f584ec510 admin admin] PATH_INFO: `/v3/projects/bfc270e7b5fe461f985c84fc087b54bc` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:48.417153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965cd10d-a004-42ab-afac-5b2f584ec510 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bfc270e7b5fe461f985c84fc087b54bc)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:48.419339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965cd10d-a004-42ab-afac-5b2f584ec510 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:48.484466 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-965cd10d-a004-42ab-afac-5b2f584ec510 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bfc270e7b5fe461f985c84fc087b54bc', 'request_id': 'req-965cd10d-a004-42ab-afac-5b2f584ec510'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:48.495479 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-965cd10d-a004-42ab-afac-5b2f584ec510 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bfc270e7b5fe461f985c84fc087b54bc', 'request_id': 'req-965cd10d-a004-42ab-afac-5b2f584ec510'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:48.521213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-965cd10d-a004-42ab-afac-5b2f584ec510 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bfc270e7b5fe461f985c84fc087b54bc', 'request_id': 'req-965cd10d-a004-42ab-afac-5b2f584ec510'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:48.534163 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-965cd10d-a004-42ab-afac-5b2f584ec510 admin admin] The token cache is being invalidate because project bfc270e7b5fe461f985c84fc087b54bc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:56:48.534484 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-965cd10d-a004-42ab-afac-5b2f584ec510 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:48.535289 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-965cd10d-a004-42ab-afac-5b2f584ec510 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:48.537215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 742/1483] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:56:48 2022] DELETE /identity/v3/projects/bfc270e7b5fe461f985c84fc087b54bc => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 00:56:48.542285 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c471c144-a0b7-48e4-ac03-d097ca081242 service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:48.554813 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c471c144-a0b7-48e4-ac03-d097ca081242 service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:48.681933 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2973eed7-f75f-4ef7-a03f-03d5c1fe4b51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:48.683104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2973eed7-f75f-4ef7-a03f-03d5c1fe4b51 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:48.683685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2973eed7-f75f-4ef7-a03f-03d5c1fe4b51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:48.684317 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2973eed7-f75f-4ef7-a03f-03d5c1fe4b51 admin admin] PATH_INFO: `/v3/users/768033c20cb643699b7824b9b619e621` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:48.693848 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2973eed7-f75f-4ef7-a03f-03d5c1fe4b51 admin admin] RBAC: Authorizing `identity:delete_user(user_id=768033c20cb643699b7824b9b619e621)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:48.696340 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2973eed7-f75f-4ef7-a03f-03d5c1fe4b51 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:48.780167 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-2973eed7-f75f-4ef7-a03f-03d5c1fe4b51 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '768033c20cb643699b7824b9b619e621'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:48.793198 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-2973eed7-f75f-4ef7-a03f-03d5c1fe4b51 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '768033c20cb643699b7824b9b619e621'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:48.794785 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-2973eed7-f75f-4ef7-a03f-03d5c1fe4b51 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '768033c20cb643699b7824b9b619e621'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:48.805174 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-2973eed7-f75f-4ef7-a03f-03d5c1fe4b51 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '768033c20cb643699b7824b9b619e621'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:48.825016 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-2973eed7-f75f-4ef7-a03f-03d5c1fe4b51 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '768033c20cb643699b7824b9b619e621'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:48.832423 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 742/1484] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:56:48 2022] DELETE /identity/v3/users/768033c20cb643699b7824b9b619e621 => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:48.857623 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965cd10d-a004-42ab-afac-5b2f584ec510 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:48.859185 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-965cd10d-a004-42ab-afac-5b2f584ec510 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:48.873706 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965cd10d-a004-42ab-afac-5b2f584ec510 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:48.952970 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b3f046-64cf-4bb7-bf6d-253b54e5bb06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:48.953624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b3f046-64cf-4bb7-bf6d-253b54e5bb06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:48.953899 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b3f046-64cf-4bb7-bf6d-253b54e5bb06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:48.954215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b3f046-64cf-4bb7-bf6d-253b54e5bb06 admin admin] PATH_INFO: `/v3/users/3877a86cf209484ca53112a98c787735` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:48.956031 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b3f046-64cf-4bb7-bf6d-253b54e5bb06 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3877a86cf209484ca53112a98c787735)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:48.958604 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b3f046-64cf-4bb7-bf6d-253b54e5bb06 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:48.978884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:49.040592 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-79b3f046-64cf-4bb7-bf6d-253b54e5bb06 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3877a86cf209484ca53112a98c787735'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:49.051836 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-79b3f046-64cf-4bb7-bf6d-253b54e5bb06 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3877a86cf209484ca53112a98c787735'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:49.053119 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-79b3f046-64cf-4bb7-bf6d-253b54e5bb06 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3877a86cf209484ca53112a98c787735'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:49.114042 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-79b3f046-64cf-4bb7-bf6d-253b54e5bb06 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3877a86cf209484ca53112a98c787735'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:49.123770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-79b3f046-64cf-4bb7-bf6d-253b54e5bb06 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3877a86cf209484ca53112a98c787735'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:49.134242 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 743/1485] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:56:48 2022] DELETE /identity/v3/users/3877a86cf209484ca53112a98c787735 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:49.139215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66327975-13fc-4843-a75a-73dee121bd4a None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:49.139498 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66327975-13fc-4843-a75a-73dee121bd4a None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:49.139799 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66327975-13fc-4843-a75a-73dee121bd4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:49.162391 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-66327975-13fc-4843-a75a-73dee121bd4a None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:49.163659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-66327975-13fc-4843-a75a-73dee121bd4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:49.194774 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:49.233257 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b3f046-64cf-4bb7-bf6d-253b54e5bb06 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:49.234806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-79b3f046-64cf-4bb7-bf6d-253b54e5bb06 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:49.240701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 743/1486] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:49.252579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b3f046-64cf-4bb7-bf6d-253b54e5bb06 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:49.252805 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66327975-13fc-4843-a75a-73dee121bd4a None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:49.254796 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-66327975-13fc-4843-a75a-73dee121bd4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:49.264406 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5abfb31-0efb-4b9c-9ce4-b055448879ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:49.265259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5abfb31-0efb-4b9c-9ce4-b055448879ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:49.265677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5abfb31-0efb-4b9c-9ce4-b055448879ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:49.266092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5abfb31-0efb-4b9c-9ce4-b055448879ad admin admin] PATH_INFO: `/v3/projects/6db9548154184b07a1186d248538f2c0` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:49.267424 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5abfb31-0efb-4b9c-9ce4-b055448879ad admin admin] RBAC: Authorizing `identity:delete_project(project_id=6db9548154184b07a1186d248538f2c0)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:49.269518 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5abfb31-0efb-4b9c-9ce4-b055448879ad admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:49.329316 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c5abfb31-0efb-4b9c-9ce4-b055448879ad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6db9548154184b07a1186d248538f2c0', 'request_id': 'req-c5abfb31-0efb-4b9c-9ce4-b055448879ad'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:49.333098 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66327975-13fc-4843-a75a-73dee121bd4a None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:49.339628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c5abfb31-0efb-4b9c-9ce4-b055448879ad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6db9548154184b07a1186d248538f2c0', 'request_id': 'req-c5abfb31-0efb-4b9c-9ce4-b055448879ad'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:49.379769 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c5abfb31-0efb-4b9c-9ce4-b055448879ad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6db9548154184b07a1186d248538f2c0', 'request_id': 'req-c5abfb31-0efb-4b9c-9ce4-b055448879ad'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:49.395685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c5abfb31-0efb-4b9c-9ce4-b055448879ad admin admin] The token cache is being invalidate because project 6db9548154184b07a1186d248538f2c0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:56:49.396004 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c5abfb31-0efb-4b9c-9ce4-b055448879ad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:49.396704 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c5abfb31-0efb-4b9c-9ce4-b055448879ad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:49.398225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 744/1487] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:56:49 2022] DELETE /identity/v3/projects/6db9548154184b07a1186d248538f2c0 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:49.402816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b094d89-dad2-4687-9504-c6ddbbc4d704 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:49.403100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b094d89-dad2-4687-9504-c6ddbbc4d704 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:49.403376 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b094d89-dad2-4687-9504-c6ddbbc4d704 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:49.414949 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f74a2a-9525-4129-a83c-90681aa60dd8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:49.415560 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f74a2a-9525-4129-a83c-90681aa60dd8 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:49.415866 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f74a2a-9525-4129-a83c-90681aa60dd8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:49.416134 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f74a2a-9525-4129-a83c-90681aa60dd8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:49.417826 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-c9f74a2a-9525-4129-a83c-90681aa60dd8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:49.424061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-2b094d89-dad2-4687-9504-c6ddbbc4d704 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:49.425291 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-2b094d89-dad2-4687-9504-c6ddbbc4d704 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:49.486206 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 745/1488] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:49.489341 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f74a2a-9525-4129-a83c-90681aa60dd8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:49.490781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f74a2a-9525-4129-a83c-90681aa60dd8 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:49.494409 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083ef6e9-2b90-4320-ad23-1ffeec695f8e None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:49.494750 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083ef6e9-2b90-4320-ad23-1ffeec695f8e None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:49.494937 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083ef6e9-2b90-4320-ad23-1ffeec695f8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:49.500449 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 744/1489] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 248 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:49.514472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-083ef6e9-2b90-4320-ad23-1ffeec695f8e None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:49.515858 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-083ef6e9-2b90-4320-ad23-1ffeec695f8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:49.528209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 746/1490] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:56:49.532594 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f74a2a-9525-4129-a83c-90681aa60dd8 service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:49.542366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f74a2a-9525-4129-a83c-90681aa60dd8 service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:49.547361 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45308f90-6989-4d40-97e5-51a28b989099 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:49.548005 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45308f90-6989-4d40-97e5-51a28b989099 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:49.548275 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45308f90-6989-4d40-97e5-51a28b989099 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:49.548539 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45308f90-6989-4d40-97e5-51a28b989099 admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:49.549178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45308f90-6989-4d40-97e5-51a28b989099 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:56:49.549467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45308f90-6989-4d40-97e5-51a28b989099 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:49.551084 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45308f90-6989-4d40-97e5-51a28b989099 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:49.564588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 745/1491] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:56:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:56:49.568927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083ef6e9-2b90-4320-ad23-1ffeec695f8e None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:49.586598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083ef6e9-2b90-4320-ad23-1ffeec695f8e None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:49.595358 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4ea667-f4ea-45a0-9da7-6d53b3ad73d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:49.596163 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4ea667-f4ea-45a0-9da7-6d53b3ad73d0 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:49.596469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4ea667-f4ea-45a0-9da7-6d53b3ad73d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:49.597235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4ea667-f4ea-45a0-9da7-6d53b3ad73d0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:49.597886 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4ea667-f4ea-45a0-9da7-6d53b3ad73d0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:49.599855 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4ea667-f4ea-45a0-9da7-6d53b3ad73d0 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:49.605331 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:49.632199 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 747/1492] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:56:49 2022] POST /identity/v3/projects => generated 381 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:56:49.636474 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45308f90-6989-4d40-97e5-51a28b989099 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:49.648327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45308f90-6989-4d40-97e5-51a28b989099 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:49.685671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:49.737686 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8c29db-de67-4a48-96aa-e75b8252fc94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:49.738312 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8c29db-de67-4a48-96aa-e75b8252fc94 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:49.738586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8c29db-de67-4a48-96aa-e75b8252fc94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:49.738853 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8c29db-de67-4a48-96aa-e75b8252fc94 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:49.739755 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8c29db-de67-4a48-96aa-e75b8252fc94 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:49.741963 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8c29db-de67-4a48-96aa-e75b8252fc94 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:49.771802 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 746/1493] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:56:49 2022] POST /identity/v3/users => generated 291 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:56:49.776197 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4ea667-f4ea-45a0-9da7-6d53b3ad73d0 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:49.787560 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4ea667-f4ea-45a0-9da7-6d53b3ad73d0 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:49.798635 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b1b998-dd25-46a9-addb-a006d188ffa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:49.799347 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b1b998-dd25-46a9-addb-a006d188ffa5 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:49.799689 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b1b998-dd25-46a9-addb-a006d188ffa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:49.800353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b1b998-dd25-46a9-addb-a006d188ffa5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:49.801305 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b1b998-dd25-46a9-addb-a006d188ffa5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:49.802945 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b1b998-dd25-46a9-addb-a006d188ffa5 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:49.822616 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 748/1494] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:49 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 00:56:49.826825 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8c29db-de67-4a48-96aa-e75b8252fc94 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:49.837739 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8c29db-de67-4a48-96aa-e75b8252fc94 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:49.842428 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-affbfa3d-d990-45c8-a55d-eccf3ff65420 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:49.843032 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affbfa3d-d990-45c8-a55d-eccf3ff65420 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:49.843307 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affbfa3d-d990-45c8-a55d-eccf3ff65420 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:49.843577 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affbfa3d-d990-45c8-a55d-eccf3ff65420 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:49.844277 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affbfa3d-d990-45c8-a55d-eccf3ff65420 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:49.845908 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affbfa3d-d990-45c8-a55d-eccf3ff65420 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:49.860370 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 747/1495] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:49 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:49.861908 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b1b998-dd25-46a9-addb-a006d188ffa5 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:49.874368 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b1b998-dd25-46a9-addb-a006d188ffa5 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:49.880189 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e93d322-f082-4240-88ec-fdafde93b451 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:49.881056 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e93d322-f082-4240-88ec-fdafde93b451 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:49.881362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e93d322-f082-4240-88ec-fdafde93b451 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:49.881737 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e93d322-f082-4240-88ec-fdafde93b451 admin admin] PATH_INFO: `/v3/projects/39308670b3024d5887ddbdfb48163a64/users/ae26fb7f51714b128d8febc67582adfb/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:49.888425 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-affbfa3d-d990-45c8-a55d-eccf3ff65420 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:49.890751 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-affbfa3d-d990-45c8-a55d-eccf3ff65420 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:49.902444 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e93d322-f082-4240-88ec-fdafde93b451 admin admin] RBAC: Authorizing `identity:create_grant(project_id=39308670b3024d5887ddbdfb48163a64, user_id=ae26fb7f51714b128d8febc67582adfb, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:49.905147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e93d322-f082-4240-88ec-fdafde93b451 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:49.919828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 749/1496] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:56:49 2022] PUT /identity/v3/projects/39308670b3024d5887ddbdfb48163a64/users/ae26fb7f51714b128d8febc67582adfb/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:49.927439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-affbfa3d-d990-45c8-a55d-eccf3ff65420 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:49.931355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4688e162-462b-44ff-8172-97f963c5400c None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:49.931640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4688e162-462b-44ff-8172-97f963c5400c None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:49.931943 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4688e162-462b-44ff-8172-97f963c5400c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:49.968495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-4688e162-462b-44ff-8172-97f963c5400c None None] MFA Rules not processed for user `ae26fb7f51714b128d8febc67582adfb`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:49.970463 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-4688e162-462b-44ff-8172-97f963c5400c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:50.000628 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b347ff1e-b567-45cc-9f47-4067484b92ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:50.001269 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b347ff1e-b567-45cc-9f47-4067484b92ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:50.001546 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b347ff1e-b567-45cc-9f47-4067484b92ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:50.001818 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b347ff1e-b567-45cc-9f47-4067484b92ae admin admin] PATH_INFO: `/v3/projects/324b3f5ee1794987b521344fd0960b2d` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:50.002952 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b347ff1e-b567-45cc-9f47-4067484b92ae admin admin] RBAC: Authorizing `identity:delete_project(project_id=324b3f5ee1794987b521344fd0960b2d)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:50.004906 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b347ff1e-b567-45cc-9f47-4067484b92ae admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:50.061243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 750/1497] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:49 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:56:50.065866 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4688e162-462b-44ff-8172-97f963c5400c None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:50.074837 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b347ff1e-b567-45cc-9f47-4067484b92ae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '324b3f5ee1794987b521344fd0960b2d', 'request_id': 'req-b347ff1e-b567-45cc-9f47-4067484b92ae'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:50.078646 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4688e162-462b-44ff-8172-97f963c5400c None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:50.081139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b347ff1e-b567-45cc-9f47-4067484b92ae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '324b3f5ee1794987b521344fd0960b2d', 'request_id': 'req-b347ff1e-b567-45cc-9f47-4067484b92ae'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:50.091340 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b347ff1e-b567-45cc-9f47-4067484b92ae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '324b3f5ee1794987b521344fd0960b2d', 'request_id': 'req-b347ff1e-b567-45cc-9f47-4067484b92ae'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:50.104164 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b347ff1e-b567-45cc-9f47-4067484b92ae admin admin] The token cache is being invalidate because project 324b3f5ee1794987b521344fd0960b2d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:56:50.104463 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b347ff1e-b567-45cc-9f47-4067484b92ae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:50.105187 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b347ff1e-b567-45cc-9f47-4067484b92ae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:50.107117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 748/1498] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:56:49 2022] DELETE /identity/v3/projects/324b3f5ee1794987b521344fd0960b2d => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 00:56:50.111006 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b347ff1e-b567-45cc-9f47-4067484b92ae admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:50.112621 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b347ff1e-b567-45cc-9f47-4067484b92ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:50.154996 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc09f84-bf6d-4306-813d-34b0037d7c66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:50.155641 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc09f84-bf6d-4306-813d-34b0037d7c66 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:50.155939 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc09f84-bf6d-4306-813d-34b0037d7c66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:50.156197 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc09f84-bf6d-4306-813d-34b0037d7c66 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:50.156876 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bc09f84-bf6d-4306-813d-34b0037d7c66 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:50.158871 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bc09f84-bf6d-4306-813d-34b0037d7c66 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:50.176090 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:50.201861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 751/1499] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:56:50 2022] POST /identity/v3/projects => generated 381 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:56:50.203206 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b347ff1e-b567-45cc-9f47-4067484b92ae admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:50.206047 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc09f84-bf6d-4306-813d-34b0037d7c66 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:50.208114 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-9bc09f84-bf6d-4306-813d-34b0037d7c66 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:50.251767 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:50.283834 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc09f84-bf6d-4306-813d-34b0037d7c66 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:50.290041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ded5815-e26d-4d4c-b9ec-eacd8105e8de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:50.291460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ded5815-e26d-4d4c-b9ec-eacd8105e8de admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:50.291835 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ded5815-e26d-4d4c-b9ec-eacd8105e8de admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:50.292139 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ded5815-e26d-4d4c-b9ec-eacd8105e8de admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:50.293797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4b5ec5-1dd7-4615-a0e0-f95fb4db1454 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:50.294600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4b5ec5-1dd7-4615-a0e0-f95fb4db1454 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:50.294994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4b5ec5-1dd7-4615-a0e0-f95fb4db1454 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:50.295354 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4b5ec5-1dd7-4615-a0e0-f95fb4db1454 admin admin] PATH_INFO: `/v3/users/6c0933c2416446e79b47344a7ba65edd` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:50.296000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ded5815-e26d-4d4c-b9ec-eacd8105e8de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:50.296876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4b5ec5-1dd7-4615-a0e0-f95fb4db1454 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6c0933c2416446e79b47344a7ba65edd)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:50.298939 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4b5ec5-1dd7-4615-a0e0-f95fb4db1454 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:50.300262 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ded5815-e26d-4d4c-b9ec-eacd8105e8de admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:50.328575 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 752/1500] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:56:50 2022] POST /identity/v3/users => generated 297 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:56:50.333308 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ded5815-e26d-4d4c-b9ec-eacd8105e8de admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:50.352554 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ded5815-e26d-4d4c-b9ec-eacd8105e8de admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:50.357773 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907f380a-25b7-4f61-8c6b-b4e340ef257f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:50.358149 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-6f4b5ec5-1dd7-4615-a0e0-f95fb4db1454 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6c0933c2416446e79b47344a7ba65edd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:50.358824 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907f380a-25b7-4f61-8c6b-b4e340ef257f admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:50.359123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907f380a-25b7-4f61-8c6b-b4e340ef257f admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:50.359470 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907f380a-25b7-4f61-8c6b-b4e340ef257f admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:50.360388 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907f380a-25b7-4f61-8c6b-b4e340ef257f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:50.368893 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907f380a-25b7-4f61-8c6b-b4e340ef257f admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:50.376338 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-6f4b5ec5-1dd7-4615-a0e0-f95fb4db1454 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6c0933c2416446e79b47344a7ba65edd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:50.377029 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-6f4b5ec5-1dd7-4615-a0e0-f95fb4db1454 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6c0933c2416446e79b47344a7ba65edd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:50.385837 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 753/1501] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:50 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 00:56:50.387352 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907f380a-25b7-4f61-8c6b-b4e340ef257f admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:50.389513 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-907f380a-25b7-4f61-8c6b-b4e340ef257f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:50.390111 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-6f4b5ec5-1dd7-4615-a0e0-f95fb4db1454 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6c0933c2416446e79b47344a7ba65edd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:50.418743 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-6f4b5ec5-1dd7-4615-a0e0-f95fb4db1454 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6c0933c2416446e79b47344a7ba65edd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:50.435849 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 749/1502] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:56:50 2022] DELETE /identity/v3/users/6c0933c2416446e79b47344a7ba65edd => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 00:56:50.450777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907f380a-25b7-4f61-8c6b-b4e340ef257f admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:50.558711 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de9e865-e9ec-4930-8131-fbd4567a003b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:50.559385 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de9e865-e9ec-4930-8131-fbd4567a003b admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:50.559659 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de9e865-e9ec-4930-8131-fbd4567a003b admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:50.559960 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de9e865-e9ec-4930-8131-fbd4567a003b admin admin] PATH_INFO: `/v3/projects/45e7045981be4c9d8a46e17b738acb99/users/6d4effde896b40a2ba836396e0d8bedd/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:50.580609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de9e865-e9ec-4930-8131-fbd4567a003b admin admin] RBAC: Authorizing `identity:create_grant(project_id=45e7045981be4c9d8a46e17b738acb99, user_id=6d4effde896b40a2ba836396e0d8bedd, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:50.585183 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de9e865-e9ec-4930-8131-fbd4567a003b admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:50.604429 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 754/1503] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:56:50 2022] PUT /identity/v3/projects/45e7045981be4c9d8a46e17b738acb99/users/6d4effde896b40a2ba836396e0d8bedd/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 00:56:50.608590 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4b5ec5-1dd7-4615-a0e0-f95fb4db1454 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:50.620478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4b5ec5-1dd7-4615-a0e0-f95fb4db1454 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:50.698183 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340a34c8-ee0f-4cac-b065-0fabf71aeaac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:50.699272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340a34c8-ee0f-4cac-b065-0fabf71aeaac admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:50.699560 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340a34c8-ee0f-4cac-b065-0fabf71aeaac admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:50.699969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340a34c8-ee0f-4cac-b065-0fabf71aeaac admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:50.700635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340a34c8-ee0f-4cac-b065-0fabf71aeaac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:50.702805 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340a34c8-ee0f-4cac-b065-0fabf71aeaac admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:50.705485 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:50.721128 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 750/1504] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:50 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:50.725362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de9e865-e9ec-4930-8131-fbd4567a003b admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:50.752774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de9e865-e9ec-4930-8131-fbd4567a003b admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:50.772945 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf4608e-19f0-44e7-b8c1-695448f94ecc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:50.773790 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf4608e-19f0-44e7-b8c1-695448f94ecc admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:50.774071 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf4608e-19f0-44e7-b8c1-695448f94ecc admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:50.774342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf4608e-19f0-44e7-b8c1-695448f94ecc admin admin] PATH_INFO: `/v3/projects/45e7045981be4c9d8a46e17b738acb99/users/6d4effde896b40a2ba836396e0d8bedd/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:50.796424 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf4608e-19f0-44e7-b8c1-695448f94ecc admin admin] RBAC: Authorizing `identity:create_grant(project_id=45e7045981be4c9d8a46e17b738acb99, user_id=6d4effde896b40a2ba836396e0d8bedd, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:50.799085 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf4608e-19f0-44e7-b8c1-695448f94ecc admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:50.835128 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 755/1505] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:56:50 2022] PUT /identity/v3/projects/45e7045981be4c9d8a46e17b738acb99/users/6d4effde896b40a2ba836396e0d8bedd/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:50.845798 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bbc347-a53e-413b-9c60-155855912c3f None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:50.846152 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bbc347-a53e-413b-9c60-155855912c3f None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:50.846519 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35bbc347-a53e-413b-9c60-155855912c3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:50.908209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-35bbc347-a53e-413b-9c60-155855912c3f None None] MFA Rules not processed for user `6d4effde896b40a2ba836396e0d8bedd`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:50.911562 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-35bbc347-a53e-413b-9c60-155855912c3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:51.005634 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 751/1506] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:50 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:51.021545 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf4608e-19f0-44e7-b8c1-695448f94ecc admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:51.023584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-bcf4608e-19f0-44e7-b8c1-695448f94ecc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:51.087780 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf4608e-19f0-44e7-b8c1-695448f94ecc admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:51.093016 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32468503-ad26-438d-a397-d9af0a796b43 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:51.093642 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32468503-ad26-438d-a397-d9af0a796b43 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:51.093954 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32468503-ad26-438d-a397-d9af0a796b43 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:51.094192 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32468503-ad26-438d-a397-d9af0a796b43 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:51.106210 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32468503-ad26-438d-a397-d9af0a796b43 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:51.107663 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32468503-ad26-438d-a397-d9af0a796b43 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:51.119448 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 756/1507] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:51 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:51.126158 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35bbc347-a53e-413b-9c60-155855912c3f None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:51.129055 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-35bbc347-a53e-413b-9c60-155855912c3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:51.243394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32468503-ad26-438d-a397-d9af0a796b43 service cinder] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:51.246632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35bbc347-a53e-413b-9c60-155855912c3f None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:51.252886 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807e9078-3aaf-4376-83d9-6238af3c922d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:51.253555 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807e9078-3aaf-4376-83d9-6238af3c922d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:51.253830 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807e9078-3aaf-4376-83d9-6238af3c922d admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:51.254174 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807e9078-3aaf-4376-83d9-6238af3c922d admin admin] PATH_INFO: `/v3/projects/450708711051450abc140191202115fd` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:51.255568 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807e9078-3aaf-4376-83d9-6238af3c922d admin admin] RBAC: Authorizing `identity:delete_project(project_id=450708711051450abc140191202115fd)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:51.257901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807e9078-3aaf-4376-83d9-6238af3c922d admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:51.262859 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32468503-ad26-438d-a397-d9af0a796b43 service cinder] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:51.267007 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178f98d3-6127-45cb-9fb5-ed9adc6deba0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:51.267617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178f98d3-6127-45cb-9fb5-ed9adc6deba0 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:51.268366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178f98d3-6127-45cb-9fb5-ed9adc6deba0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:51.268632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178f98d3-6127-45cb-9fb5-ed9adc6deba0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:51.271160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-178f98d3-6127-45cb-9fb5-ed9adc6deba0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:51.289101 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:51.342249 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-807e9078-3aaf-4376-83d9-6238af3c922d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '450708711051450abc140191202115fd', 'request_id': 'req-807e9078-3aaf-4376-83d9-6238af3c922d'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:51.349280 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-807e9078-3aaf-4376-83d9-6238af3c922d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '450708711051450abc140191202115fd', 'request_id': 'req-807e9078-3aaf-4376-83d9-6238af3c922d'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:51.360538 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178f98d3-6127-45cb-9fb5-ed9adc6deba0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:51.361634 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-807e9078-3aaf-4376-83d9-6238af3c922d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '450708711051450abc140191202115fd', 'request_id': 'req-807e9078-3aaf-4376-83d9-6238af3c922d'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:51.364823 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178f98d3-6127-45cb-9fb5-ed9adc6deba0 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:51.378382 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-807e9078-3aaf-4376-83d9-6238af3c922d admin admin] The token cache is being invalidate because project 450708711051450abc140191202115fd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:56:51.378693 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-807e9078-3aaf-4376-83d9-6238af3c922d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:51.378830 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 757/1508] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:51 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 00:56:51.380126 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-807e9078-3aaf-4376-83d9-6238af3c922d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:51.382011 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 752/1509] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:56:51 2022] DELETE /identity/v3/projects/450708711051450abc140191202115fd => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:51.387409 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd57f13-47ba-49a9-8415-b9086edf5c18 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:51.387647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd57f13-47ba-49a9-8415-b9086edf5c18 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:51.387944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd57f13-47ba-49a9-8415-b9086edf5c18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:51.415798 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-0cd57f13-47ba-49a9-8415-b9086edf5c18 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:51.417349 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-0cd57f13-47ba-49a9-8415-b9086edf5c18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:51.487751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 758/1510] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:51.494428 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a51af9e-cb9b-4219-a29b-821af745b11f None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:51.494804 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a51af9e-cb9b-4219-a29b-821af745b11f None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:51.495089 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a51af9e-cb9b-4219-a29b-821af745b11f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:51.523142 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-2a51af9e-cb9b-4219-a29b-821af745b11f None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:51.523142 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-2a51af9e-cb9b-4219-a29b-821af745b11f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:51.533580 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 753/1511] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:51.536090 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd57f13-47ba-49a9-8415-b9086edf5c18 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:51.546415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd57f13-47ba-49a9-8415-b9086edf5c18 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:51.551215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68623286-91f6-44ba-bc0a-dd3b2a970b55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:51.552227 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68623286-91f6-44ba-bc0a-dd3b2a970b55 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:51.552227 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68623286-91f6-44ba-bc0a-dd3b2a970b55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:51.552495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68623286-91f6-44ba-bc0a-dd3b2a970b55 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:51.553105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68623286-91f6-44ba-bc0a-dd3b2a970b55 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:56:51.553550 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68623286-91f6-44ba-bc0a-dd3b2a970b55 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:51.555793 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68623286-91f6-44ba-bc0a-dd3b2a970b55 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:51.575847 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 759/1512] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:56:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:56:51.577431 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a51af9e-cb9b-4219-a29b-821af745b11f None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:51.591210 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a51af9e-cb9b-4219-a29b-821af745b11f None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:51.596008 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f73a6a0-c1ae-4979-b543-ccd45048cbff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:51.596611 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f73a6a0-c1ae-4979-b543-ccd45048cbff admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:51.596892 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f73a6a0-c1ae-4979-b543-ccd45048cbff admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:51.597166 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f73a6a0-c1ae-4979-b543-ccd45048cbff admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:51.597794 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f73a6a0-c1ae-4979-b543-ccd45048cbff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:51.599692 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f73a6a0-c1ae-4979-b543-ccd45048cbff admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:51.604564 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:51.637846 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 754/1513] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:56:51 2022] POST /identity/v3/projects => generated 377 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:56:51.642440 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68623286-91f6-44ba-bc0a-dd3b2a970b55 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:51.652560 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68623286-91f6-44ba-bc0a-dd3b2a970b55 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:51.681393 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f73a6a0-c1ae-4979-b543-ccd45048cbff admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:51.683645 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-4f73a6a0-c1ae-4979-b543-ccd45048cbff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:51.715358 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5218839b-2598-427c-b861-03f062d3b9dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:51.716020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5218839b-2598-427c-b861-03f062d3b9dd admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:51.716320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5218839b-2598-427c-b861-03f062d3b9dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:51.716586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5218839b-2598-427c-b861-03f062d3b9dd admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:51.717208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5218839b-2598-427c-b861-03f062d3b9dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:51.719290 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5218839b-2598-427c-b861-03f062d3b9dd admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:51.741740 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 760/1514] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:56:51 2022] POST /identity/v3/users => generated 289 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:56:51.745722 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5218839b-2598-427c-b861-03f062d3b9dd admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:51.752892 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f73a6a0-c1ae-4979-b543-ccd45048cbff admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:51.755227 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5218839b-2598-427c-b861-03f062d3b9dd admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:51.758329 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac175909-7498-45f1-ab09-daae4f256415 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:51.759280 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac175909-7498-45f1-ab09-daae4f256415 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:51.759542 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac175909-7498-45f1-ab09-daae4f256415 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:51.759876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac175909-7498-45f1-ab09-daae4f256415 admin admin] PATH_INFO: `/v3/users/ae26fb7f51714b128d8febc67582adfb` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:51.760016 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5666a5a6-7559-42c1-a710-9aa0fe6c0191 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:51.760596 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5666a5a6-7559-42c1-a710-9aa0fe6c0191 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:51.760869 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5666a5a6-7559-42c1-a710-9aa0fe6c0191 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:51.761135 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5666a5a6-7559-42c1-a710-9aa0fe6c0191 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:51.761236 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac175909-7498-45f1-ab09-daae4f256415 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ae26fb7f51714b128d8febc67582adfb)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:51.761759 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5666a5a6-7559-42c1-a710-9aa0fe6c0191 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:51.763136 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac175909-7498-45f1-ab09-daae4f256415 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:51.763368 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5666a5a6-7559-42c1-a710-9aa0fe6c0191 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:51.778751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 761/1515] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:51 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 00:56:51.779962 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5666a5a6-7559-42c1-a710-9aa0fe6c0191 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:51.789945 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5666a5a6-7559-42c1-a710-9aa0fe6c0191 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:51.796009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456c9fd9-6b8e-4922-9fdc-d56c4c0b2de6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:51.796807 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456c9fd9-6b8e-4922-9fdc-d56c4c0b2de6 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:51.797083 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456c9fd9-6b8e-4922-9fdc-d56c4c0b2de6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:51.797818 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456c9fd9-6b8e-4922-9fdc-d56c4c0b2de6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:51.798528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456c9fd9-6b8e-4922-9fdc-d56c4c0b2de6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:51.800468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456c9fd9-6b8e-4922-9fdc-d56c4c0b2de6 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:51.812047 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 762/1516] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:51 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 00:56:51.815423 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456c9fd9-6b8e-4922-9fdc-d56c4c0b2de6 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:51.825552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-ac175909-7498-45f1-ab09-daae4f256415 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ae26fb7f51714b128d8febc67582adfb'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:51.826332 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456c9fd9-6b8e-4922-9fdc-d56c4c0b2de6 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:51.833336 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3464e85e-28c9-4998-9585-0ad18ccbd133 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:51.834159 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3464e85e-28c9-4998-9585-0ad18ccbd133 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:51.834419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3464e85e-28c9-4998-9585-0ad18ccbd133 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:51.834767 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3464e85e-28c9-4998-9585-0ad18ccbd133 admin admin] PATH_INFO: `/v3/projects/5f9254764af44e9dbee0907067094154/users/ffcee427ba844256a2533311ab7f4963/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:51.837330 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-ac175909-7498-45f1-ab09-daae4f256415 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ae26fb7f51714b128d8febc67582adfb'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:51.838320 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-ac175909-7498-45f1-ab09-daae4f256415 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ae26fb7f51714b128d8febc67582adfb'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:51.840053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:51.847512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-ac175909-7498-45f1-ab09-daae4f256415 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ae26fb7f51714b128d8febc67582adfb'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:51.860448 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3464e85e-28c9-4998-9585-0ad18ccbd133 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f9254764af44e9dbee0907067094154, user_id=ffcee427ba844256a2533311ab7f4963, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:51.863434 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3464e85e-28c9-4998-9585-0ad18ccbd133 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:51.866045 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-ac175909-7498-45f1-ab09-daae4f256415 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ae26fb7f51714b128d8febc67582adfb'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:51.873322 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 755/1517] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:56:51 2022] DELETE /identity/v3/users/ae26fb7f51714b128d8febc67582adfb => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:51.877631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555dfdaf-41b8-4be8-83fe-571b9ebaf3c5 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:51.878167 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555dfdaf-41b8-4be8-83fe-571b9ebaf3c5 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:51.878582 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555dfdaf-41b8-4be8-83fe-571b9ebaf3c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:51.880985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 763/1518] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:56:51 2022] PUT /identity/v3/projects/5f9254764af44e9dbee0907067094154/users/ffcee427ba844256a2533311ab7f4963/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:51.892747 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed6a173-c99f-47d8-a150-f4d1455f3af0 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:51.893243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed6a173-c99f-47d8-a150-f4d1455f3af0 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:51.893521 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed6a173-c99f-47d8-a150-f4d1455f3af0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:51.901863 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-555dfdaf-41b8-4be8-83fe-571b9ebaf3c5 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:51.903072 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-555dfdaf-41b8-4be8-83fe-571b9ebaf3c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:51.924882 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-fed6a173-c99f-47d8-a150-f4d1455f3af0 None None] MFA Rules not processed for user `ffcee427ba844256a2533311ab7f4963`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:51.926085 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-fed6a173-c99f-47d8-a150-f4d1455f3af0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:51.987655 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 756/1519] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:56:52.006584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555dfdaf-41b8-4be8-83fe-571b9ebaf3c5 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:52.008319 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-555dfdaf-41b8-4be8-83fe-571b9ebaf3c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:52.017129 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:52.102590 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555dfdaf-41b8-4be8-83fe-571b9ebaf3c5 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:52.111860 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 764/1520] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:51 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:52.112421 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f004cb4-a59e-4b00-86d5-2338cdccd333 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:52.113047 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f004cb4-a59e-4b00-86d5-2338cdccd333 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:52.113318 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f004cb4-a59e-4b00-86d5-2338cdccd333 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:52.113708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f004cb4-a59e-4b00-86d5-2338cdccd333 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:52.115521 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed6a173-c99f-47d8-a150-f4d1455f3af0 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:52.117550 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-fed6a173-c99f-47d8-a150-f4d1455f3af0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:52.144878 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f004cb4-a59e-4b00-86d5-2338cdccd333 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:52.146488 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f004cb4-a59e-4b00-86d5-2338cdccd333 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:52.150312 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed6a173-c99f-47d8-a150-f4d1455f3af0 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:52.176295 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 757/1521] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:52.188001 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd98e78-467b-4b30-adf9-67b3ca752e03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:52.190345 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd98e78-467b-4b30-adf9-67b3ca752e03 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:52.190629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd98e78-467b-4b30-adf9-67b3ca752e03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:52.191054 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd98e78-467b-4b30-adf9-67b3ca752e03 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:52.191711 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd98e78-467b-4b30-adf9-67b3ca752e03 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:52.193838 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd98e78-467b-4b30-adf9-67b3ca752e03 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:52.240935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 765/1522] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:56:52 2022] POST /identity/v3/projects => generated 377 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:56:52.244818 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f004cb4-a59e-4b00-86d5-2338cdccd333 service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:52.258165 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f004cb4-a59e-4b00-86d5-2338cdccd333 service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:52.373519 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55eaf6c-0559-4e86-8eae-04b6a9d2342b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:52.374299 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55eaf6c-0559-4e86-8eae-04b6a9d2342b admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:52.374568 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55eaf6c-0559-4e86-8eae-04b6a9d2342b admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:52.374900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55eaf6c-0559-4e86-8eae-04b6a9d2342b admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:52.375615 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55eaf6c-0559-4e86-8eae-04b6a9d2342b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:52.377823 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55eaf6c-0559-4e86-8eae-04b6a9d2342b admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:52.408361 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 758/1523] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:56:52 2022] POST /identity/v3/users => generated 295 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:56:52.413077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd98e78-467b-4b30-adf9-67b3ca752e03 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:52.425328 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd98e78-467b-4b30-adf9-67b3ca752e03 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:52.430961 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed27e196-5aae-4ac1-966f-6931618d979c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:52.431823 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed27e196-5aae-4ac1-966f-6931618d979c admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:52.432160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed27e196-5aae-4ac1-966f-6931618d979c admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:52.432634 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed27e196-5aae-4ac1-966f-6931618d979c admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:52.433150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed27e196-5aae-4ac1-966f-6931618d979c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:52.435087 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed27e196-5aae-4ac1-966f-6931618d979c admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:52.447909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 766/1524] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:52 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:52.452497 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55eaf6c-0559-4e86-8eae-04b6a9d2342b admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:52.466815 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55eaf6c-0559-4e86-8eae-04b6a9d2342b admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:52.472491 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0bcb88-be86-4805-83e2-b2afcb9c15b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:52.473185 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0bcb88-be86-4805-83e2-b2afcb9c15b6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:52.473469 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0bcb88-be86-4805-83e2-b2afcb9c15b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:52.473730 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0bcb88-be86-4805-83e2-b2afcb9c15b6 admin admin] PATH_INFO: `/v3/projects/8435251b59f742dfa7c4366d91a7c10d/users/329fbcbc1e824a6aa5fe4f4eab099243/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:52.496289 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0bcb88-be86-4805-83e2-b2afcb9c15b6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8435251b59f742dfa7c4366d91a7c10d, user_id=329fbcbc1e824a6aa5fe4f4eab099243, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:52.498995 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0bcb88-be86-4805-83e2-b2afcb9c15b6 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:52.514229 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 759/1525] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:56:52 2022] PUT /identity/v3/projects/8435251b59f742dfa7c4366d91a7c10d/users/329fbcbc1e824a6aa5fe4f4eab099243/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:52.518280 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed27e196-5aae-4ac1-966f-6931618d979c admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:52.529950 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed27e196-5aae-4ac1-966f-6931618d979c admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:52.567184 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0bcb88-be86-4805-83e2-b2afcb9c15b6 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:52.570972 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-fc0bcb88-be86-4805-83e2-b2afcb9c15b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:52.601369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656f1e22-53cf-49a0-a3b5-5669f6ac5624 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:52.602789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656f1e22-53cf-49a0-a3b5-5669f6ac5624 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:52.603071 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656f1e22-53cf-49a0-a3b5-5669f6ac5624 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:52.603343 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656f1e22-53cf-49a0-a3b5-5669f6ac5624 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:52.604093 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656f1e22-53cf-49a0-a3b5-5669f6ac5624 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:52.607902 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656f1e22-53cf-49a0-a3b5-5669f6ac5624 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:52.620864 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 767/1526] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:52 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:52.622130 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656f1e22-53cf-49a0-a3b5-5669f6ac5624 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:52.634524 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656f1e22-53cf-49a0-a3b5-5669f6ac5624 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:52.641413 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c975120b-325d-4026-9fee-6d89ca76b5cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:52.641413 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c975120b-325d-4026-9fee-6d89ca76b5cc admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:52.641413 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c975120b-325d-4026-9fee-6d89ca76b5cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:52.641413 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c975120b-325d-4026-9fee-6d89ca76b5cc admin admin] PATH_INFO: `/v3/projects/8435251b59f742dfa7c4366d91a7c10d/users/329fbcbc1e824a6aa5fe4f4eab099243/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:52.654590 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c975120b-325d-4026-9fee-6d89ca76b5cc admin admin] RBAC: Authorizing `identity:create_grant(project_id=8435251b59f742dfa7c4366d91a7c10d, user_id=329fbcbc1e824a6aa5fe4f4eab099243, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:52.654590 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c975120b-325d-4026-9fee-6d89ca76b5cc admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:52.656700 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0bcb88-be86-4805-83e2-b2afcb9c15b6 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:52.662327 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 768/1527] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:56:52 2022] PUT /identity/v3/projects/8435251b59f742dfa7c4366d91a7c10d/users/329fbcbc1e824a6aa5fe4f4eab099243/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:52.673579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08dcef4-8a9f-4e74-8ea4-5e1de821269e None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:52.673856 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08dcef4-8a9f-4e74-8ea4-5e1de821269e None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:52.674131 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08dcef4-8a9f-4e74-8ea4-5e1de821269e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:52.675814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:52.706079 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-c08dcef4-8a9f-4e74-8ea4-5e1de821269e None None] MFA Rules not processed for user `329fbcbc1e824a6aa5fe4f4eab099243`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:52.707372 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c08dcef4-8a9f-4e74-8ea4-5e1de821269e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:52.719489 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:52.719758 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147a1c55-d80b-48e3-92a0-2e82c849f653 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:52.720459 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147a1c55-d80b-48e3-92a0-2e82c849f653 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:52.720979 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147a1c55-d80b-48e3-92a0-2e82c849f653 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:52.720979 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147a1c55-d80b-48e3-92a0-2e82c849f653 admin admin] PATH_INFO: `/v3/projects/39308670b3024d5887ddbdfb48163a64` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:52.722267 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147a1c55-d80b-48e3-92a0-2e82c849f653 admin admin] RBAC: Authorizing `identity:delete_project(project_id=39308670b3024d5887ddbdfb48163a64)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:52.724237 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147a1c55-d80b-48e3-92a0-2e82c849f653 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:52.774889 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-147a1c55-d80b-48e3-92a0-2e82c849f653 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '39308670b3024d5887ddbdfb48163a64', 'request_id': 'req-147a1c55-d80b-48e3-92a0-2e82c849f653'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:52.780528 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-147a1c55-d80b-48e3-92a0-2e82c849f653 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '39308670b3024d5887ddbdfb48163a64', 'request_id': 'req-147a1c55-d80b-48e3-92a0-2e82c849f653'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:52.781742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 769/1528] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:52 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:56:52.789651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-147a1c55-d80b-48e3-92a0-2e82c849f653 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '39308670b3024d5887ddbdfb48163a64', 'request_id': 'req-147a1c55-d80b-48e3-92a0-2e82c849f653'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:52.793671 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08dcef4-8a9f-4e74-8ea4-5e1de821269e None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:52.795121 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c08dcef4-8a9f-4e74-8ea4-5e1de821269e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:52.802744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-147a1c55-d80b-48e3-92a0-2e82c849f653 admin admin] The token cache is being invalidate because project 39308670b3024d5887ddbdfb48163a64 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:56:52.803070 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-147a1c55-d80b-48e3-92a0-2e82c849f653 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:52.804048 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-147a1c55-d80b-48e3-92a0-2e82c849f653 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:52.810655 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 760/1529] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:56:52 2022] DELETE /identity/v3/projects/39308670b3024d5887ddbdfb48163a64 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 00:56:52.810655 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147a1c55-d80b-48e3-92a0-2e82c849f653 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:52.812640 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-147a1c55-d80b-48e3-92a0-2e82c849f653 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:52.859391 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08dcef4-8a9f-4e74-8ea4-5e1de821269e None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:52.877366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147a1c55-d80b-48e3-92a0-2e82c849f653 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:52.882255 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62278ab-7c4b-408b-82b6-c0346588608a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:52.882899 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62278ab-7c4b-408b-82b6-c0346588608a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:52.883171 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62278ab-7c4b-408b-82b6-c0346588608a admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:52.883432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62278ab-7c4b-408b-82b6-c0346588608a admin admin] PATH_INFO: `/v3/users/6d4effde896b40a2ba836396e0d8bedd` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:52.884661 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62278ab-7c4b-408b-82b6-c0346588608a admin admin] RBAC: Authorizing `identity:delete_user(user_id=6d4effde896b40a2ba836396e0d8bedd)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:52.886591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62278ab-7c4b-408b-82b6-c0346588608a admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:52.908480 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798c240a-aeb2-4495-ac47-812f53f396ef service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:52.909099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798c240a-aeb2-4495-ac47-812f53f396ef service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:52.909372 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798c240a-aeb2-4495-ac47-812f53f396ef service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:52.909633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798c240a-aeb2-4495-ac47-812f53f396ef service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:52.911374 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-798c240a-aeb2-4495-ac47-812f53f396ef service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:52.945863 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a62278ab-7c4b-408b-82b6-c0346588608a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6d4effde896b40a2ba836396e0d8bedd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:52.951781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:52.957098 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a62278ab-7c4b-408b-82b6-c0346588608a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6d4effde896b40a2ba836396e0d8bedd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:52.957935 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a62278ab-7c4b-408b-82b6-c0346588608a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6d4effde896b40a2ba836396e0d8bedd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:52.968446 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a62278ab-7c4b-408b-82b6-c0346588608a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6d4effde896b40a2ba836396e0d8bedd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:52.986416 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a62278ab-7c4b-408b-82b6-c0346588608a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6d4effde896b40a2ba836396e0d8bedd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:52.993344 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798c240a-aeb2-4495-ac47-812f53f396ef service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:52.993499 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 761/1530] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:56:52 2022] DELETE /identity/v3/users/6d4effde896b40a2ba836396e0d8bedd => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 00:56:52.994991 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798c240a-aeb2-4495-ac47-812f53f396ef service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:53.005474 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 770/1531] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:52 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 212 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 00:56:53.443293 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62278ab-7c4b-408b-82b6-c0346588608a admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:53.446419 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a62278ab-7c4b-408b-82b6-c0346588608a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:53.606365 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62278ab-7c4b-408b-82b6-c0346588608a admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:53.611395 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881549c0-5a7e-4265-bb3a-6d52b9c3bd7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:53.612122 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881549c0-5a7e-4265-bb3a-6d52b9c3bd7b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:53.612397 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881549c0-5a7e-4265-bb3a-6d52b9c3bd7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:53.612667 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881549c0-5a7e-4265-bb3a-6d52b9c3bd7b admin admin] PATH_INFO: `/v3/projects/45e7045981be4c9d8a46e17b738acb99` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:53.613774 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881549c0-5a7e-4265-bb3a-6d52b9c3bd7b admin admin] RBAC: Authorizing `identity:delete_project(project_id=45e7045981be4c9d8a46e17b738acb99)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:53.615700 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881549c0-5a7e-4265-bb3a-6d52b9c3bd7b admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:53.667850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-881549c0-5a7e-4265-bb3a-6d52b9c3bd7b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '45e7045981be4c9d8a46e17b738acb99', 'request_id': 'req-881549c0-5a7e-4265-bb3a-6d52b9c3bd7b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:53.682131 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-881549c0-5a7e-4265-bb3a-6d52b9c3bd7b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '45e7045981be4c9d8a46e17b738acb99', 'request_id': 'req-881549c0-5a7e-4265-bb3a-6d52b9c3bd7b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:53.692618 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-881549c0-5a7e-4265-bb3a-6d52b9c3bd7b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '45e7045981be4c9d8a46e17b738acb99', 'request_id': 'req-881549c0-5a7e-4265-bb3a-6d52b9c3bd7b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:53.707054 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-881549c0-5a7e-4265-bb3a-6d52b9c3bd7b admin admin] The token cache is being invalidate because project 45e7045981be4c9d8a46e17b738acb99 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:56:53.707539 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-881549c0-5a7e-4265-bb3a-6d52b9c3bd7b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:53.708414 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-881549c0-5a7e-4265-bb3a-6d52b9c3bd7b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:53.710557 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 762/1532] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:56:53 2022] DELETE /identity/v3/projects/45e7045981be4c9d8a46e17b738acb99 => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:53.715323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87da723a-4231-448c-8217-a751055c9cfb None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:53.715842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87da723a-4231-448c-8217-a751055c9cfb None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:53.715941 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87da723a-4231-448c-8217-a751055c9cfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:53.736968 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-87da723a-4231-448c-8217-a751055c9cfb None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:53.738278 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-87da723a-4231-448c-8217-a751055c9cfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:53.752202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:53.802565 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 771/1533] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:53.810914 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cf454f-69bb-45c9-b3cf-db39de3ed89d None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:53.811273 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cf454f-69bb-45c9-b3cf-db39de3ed89d None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:53.811604 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cf454f-69bb-45c9-b3cf-db39de3ed89d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:53.831946 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-29cf454f-69bb-45c9-b3cf-db39de3ed89d None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:53.833216 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-29cf454f-69bb-45c9-b3cf-db39de3ed89d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:53.841007 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 763/1534] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:53.845394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87da723a-4231-448c-8217-a751055c9cfb None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:53.855933 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87da723a-4231-448c-8217-a751055c9cfb None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:53.861469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2da1c3-be4b-4948-bcfe-26912a212b95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:53.862190 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2da1c3-be4b-4948-bcfe-26912a212b95 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:53.862526 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2da1c3-be4b-4948-bcfe-26912a212b95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:53.862855 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2da1c3-be4b-4948-bcfe-26912a212b95 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:53.863597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2da1c3-be4b-4948-bcfe-26912a212b95 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:56:53.863936 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2da1c3-be4b-4948-bcfe-26912a212b95 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:53.865544 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2da1c3-be4b-4948-bcfe-26912a212b95 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:53.878611 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 772/1535] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:56:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:56:53.884011 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29cf454f-69bb-45c9-b3cf-db39de3ed89d None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:53.893483 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29cf454f-69bb-45c9-b3cf-db39de3ed89d None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:53.898329 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0bcd17-6f30-4012-82be-35535be55f04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:53.899023 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0bcd17-6f30-4012-82be-35535be55f04 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:53.899354 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0bcd17-6f30-4012-82be-35535be55f04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:53.899677 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0bcd17-6f30-4012-82be-35535be55f04 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:53.900386 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0bcd17-6f30-4012-82be-35535be55f04 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:53.902347 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0bcd17-6f30-4012-82be-35535be55f04 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:53.933296 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 764/1536] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:56:53 2022] POST /identity/v3/projects => generated 367 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:56:53.937883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2da1c3-be4b-4948-bcfe-26912a212b95 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:53.949002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2da1c3-be4b-4948-bcfe-26912a212b95 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:54.012628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b700d798-dc97-4bfa-ae6f-63b34685bea6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:54.013281 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b700d798-dc97-4bfa-ae6f-63b34685bea6 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:54.013709 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b700d798-dc97-4bfa-ae6f-63b34685bea6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:54.013976 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b700d798-dc97-4bfa-ae6f-63b34685bea6 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:54.014605 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b700d798-dc97-4bfa-ae6f-63b34685bea6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:54.016749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b700d798-dc97-4bfa-ae6f-63b34685bea6 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:54.038870 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 773/1537] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:56:53 2022] POST /identity/v3/users => generated 284 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:56:54.043435 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0bcd17-6f30-4012-82be-35535be55f04 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:54.056266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0bcd17-6f30-4012-82be-35535be55f04 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:54.062138 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44b519c-a8f3-43ce-ab26-fca6824e05cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:54.062754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44b519c-a8f3-43ce-ab26-fca6824e05cb admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:54.063023 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44b519c-a8f3-43ce-ab26-fca6824e05cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:54.063285 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44b519c-a8f3-43ce-ab26-fca6824e05cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:54.063968 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44b519c-a8f3-43ce-ab26-fca6824e05cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:54.065576 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44b519c-a8f3-43ce-ab26-fca6824e05cb admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:54.076857 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 765/1538] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:54 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:54.081103 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b700d798-dc97-4bfa-ae6f-63b34685bea6 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:54.091523 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b700d798-dc97-4bfa-ae6f-63b34685bea6 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:54.096266 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf91b1f-6163-467b-ab3d-edd4da0f2b5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:54.096897 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf91b1f-6163-467b-ab3d-edd4da0f2b5b admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:54.097174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf91b1f-6163-467b-ab3d-edd4da0f2b5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:54.097443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf91b1f-6163-467b-ab3d-edd4da0f2b5b admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:54.098110 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf91b1f-6163-467b-ab3d-edd4da0f2b5b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:54.099950 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf91b1f-6163-467b-ab3d-edd4da0f2b5b admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:54.111136 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 774/1539] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:54 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:54.115518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44b519c-a8f3-43ce-ab26-fca6824e05cb admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:54.126395 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44b519c-a8f3-43ce-ab26-fca6824e05cb admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:54.134809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7e74f0-1ee2-497f-9f43-7490522b5b35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:54.135753 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7e74f0-1ee2-497f-9f43-7490522b5b35 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:54.136332 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7e74f0-1ee2-497f-9f43-7490522b5b35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:54.136572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7e74f0-1ee2-497f-9f43-7490522b5b35 admin admin] PATH_INFO: `/v3/projects/574f30ddffde423cb6ba7c57f1d6bb79/users/803761c6461e47148dfd74040f46f691/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:54.173523 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf7e74f0-1ee2-497f-9f43-7490522b5b35 admin admin] RBAC: Authorizing `identity:create_grant(project_id=574f30ddffde423cb6ba7c57f1d6bb79, user_id=803761c6461e47148dfd74040f46f691, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:54.177777 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf7e74f0-1ee2-497f-9f43-7490522b5b35 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:54.181437 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=88,keystone:DELETE=9,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:54.194732 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 766/1540] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:56:54 2022] PUT /identity/v3/projects/574f30ddffde423cb6ba7c57f1d6bb79/users/803761c6461e47148dfd74040f46f691/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:54.206273 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dccaf9-f611-4843-ab84-4f3da055263f None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:54.206558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dccaf9-f611-4843-ab84-4f3da055263f None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:54.206821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dccaf9-f611-4843-ab84-4f3da055263f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:54.234413 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-e9dccaf9-f611-4843-ab84-4f3da055263f None None] MFA Rules not processed for user `803761c6461e47148dfd74040f46f691`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:54.235659 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e9dccaf9-f611-4843-ab84-4f3da055263f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:54.329092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 775/1541] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:54 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:54.343758 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7e74f0-1ee2-497f-9f43-7490522b5b35 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:54.344319 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-bf7e74f0-1ee2-497f-9f43-7490522b5b35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:54.440562 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7e74f0-1ee2-497f-9f43-7490522b5b35 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:54.446454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b058e38e-e1f1-4aeb-bcdb-be94b7c9517b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:54.449230 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b058e38e-e1f1-4aeb-bcdb-be94b7c9517b service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:54.449230 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b058e38e-e1f1-4aeb-bcdb-be94b7c9517b service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:54.449230 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b058e38e-e1f1-4aeb-bcdb-be94b7c9517b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:54.460721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b058e38e-e1f1-4aeb-bcdb-be94b7c9517b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:54.462505 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b058e38e-e1f1-4aeb-bcdb-be94b7c9517b service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:54.494591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 767/1542] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:54 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:54.495907 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dccaf9-f611-4843-ab84-4f3da055263f None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:54.518101 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:54.589356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dccaf9-f611-4843-ab84-4f3da055263f None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:54.594775 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fb6640-0143-4fc1-912f-0c930df9b95e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:54.596140 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fb6640-0143-4fc1-912f-0c930df9b95e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:54.596140 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fb6640-0143-4fc1-912f-0c930df9b95e admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:54.596356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fb6640-0143-4fc1-912f-0c930df9b95e admin admin] PATH_INFO: `/v3/users/803761c6461e47148dfd74040f46f691` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:54.597822 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79fb6640-0143-4fc1-912f-0c930df9b95e admin admin] RBAC: Authorizing `identity:delete_user(user_id=803761c6461e47148dfd74040f46f691)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:54.601826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79fb6640-0143-4fc1-912f-0c930df9b95e admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:54.825579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-79fb6640-0143-4fc1-912f-0c930df9b95e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '803761c6461e47148dfd74040f46f691'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:54.834220 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-79fb6640-0143-4fc1-912f-0c930df9b95e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '803761c6461e47148dfd74040f46f691'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:54.835102 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-79fb6640-0143-4fc1-912f-0c930df9b95e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '803761c6461e47148dfd74040f46f691'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:54.852926 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-79fb6640-0143-4fc1-912f-0c930df9b95e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '803761c6461e47148dfd74040f46f691'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:54.870052 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-79fb6640-0143-4fc1-912f-0c930df9b95e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '803761c6461e47148dfd74040f46f691'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:54.879869 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 776/1543] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:56:54 2022] DELETE /identity/v3/users/803761c6461e47148dfd74040f46f691 => generated 0 bytes in 385 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:54.885259 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1990b7bc-8298-4d8b-87a5-0f6d85dfec2a None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:54.885754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1990b7bc-8298-4d8b-87a5-0f6d85dfec2a None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:54.887092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1990b7bc-8298-4d8b-87a5-0f6d85dfec2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:54.909997 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-1990b7bc-8298-4d8b-87a5-0f6d85dfec2a None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:54.911751 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-1990b7bc-8298-4d8b-87a5-0f6d85dfec2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:54.981178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 768/1544] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:56:55.001716 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fb6640-0143-4fc1-912f-0c930df9b95e admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:55.002907 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-79fb6640-0143-4fc1-912f-0c930df9b95e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:55.086759 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fb6640-0143-4fc1-912f-0c930df9b95e admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:55.092599 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c887fe43-fe03-4ad5-912b-5e6c1ad37f50 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:55.093479 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c887fe43-fe03-4ad5-912b-5e6c1ad37f50 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:55.094225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c887fe43-fe03-4ad5-912b-5e6c1ad37f50 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:55.094689 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c887fe43-fe03-4ad5-912b-5e6c1ad37f50 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:55.113099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c887fe43-fe03-4ad5-912b-5e6c1ad37f50 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:55.114719 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c887fe43-fe03-4ad5-912b-5e6c1ad37f50 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:55.126870 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 777/1545] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:56:55.621736 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1990b7bc-8298-4d8b-87a5-0f6d85dfec2a None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:55.623388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-1990b7bc-8298-4d8b-87a5-0f6d85dfec2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:55.637272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1990b7bc-8298-4d8b-87a5-0f6d85dfec2a None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:55.642178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e291f8c3-4c2d-4396-ba0e-95e6048d1a79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:55.642829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e291f8c3-4c2d-4396-ba0e-95e6048d1a79 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:55.643105 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e291f8c3-4c2d-4396-ba0e-95e6048d1a79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:55.643517 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e291f8c3-4c2d-4396-ba0e-95e6048d1a79 admin admin] PATH_INFO: `/v3/projects/574f30ddffde423cb6ba7c57f1d6bb79` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:55.644533 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e291f8c3-4c2d-4396-ba0e-95e6048d1a79 admin admin] RBAC: Authorizing `identity:delete_project(project_id=574f30ddffde423cb6ba7c57f1d6bb79)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:55.646645 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e291f8c3-4c2d-4396-ba0e-95e6048d1a79 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:55.703064 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-e291f8c3-4c2d-4396-ba0e-95e6048d1a79 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '574f30ddffde423cb6ba7c57f1d6bb79', 'request_id': 'req-e291f8c3-4c2d-4396-ba0e-95e6048d1a79'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:55.709117 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-e291f8c3-4c2d-4396-ba0e-95e6048d1a79 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '574f30ddffde423cb6ba7c57f1d6bb79', 'request_id': 'req-e291f8c3-4c2d-4396-ba0e-95e6048d1a79'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:55.719189 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-e291f8c3-4c2d-4396-ba0e-95e6048d1a79 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '574f30ddffde423cb6ba7c57f1d6bb79', 'request_id': 'req-e291f8c3-4c2d-4396-ba0e-95e6048d1a79'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:55.733781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-e291f8c3-4c2d-4396-ba0e-95e6048d1a79 admin admin] The token cache is being invalidate because project 574f30ddffde423cb6ba7c57f1d6bb79 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:56:55.734076 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-e291f8c3-4c2d-4396-ba0e-95e6048d1a79 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:55.734800 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-e291f8c3-4c2d-4396-ba0e-95e6048d1a79 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:56:55.736405 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 769/1546] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:56:55 2022] DELETE /identity/v3/projects/574f30ddffde423cb6ba7c57f1d6bb79 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:55.741347 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677d6898-9f7b-4459-9732-3049f3dc47b8 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:55.741733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677d6898-9f7b-4459-9732-3049f3dc47b8 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:55.742015 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677d6898-9f7b-4459-9732-3049f3dc47b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:55.761602 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-677d6898-9f7b-4459-9732-3049f3dc47b8 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:55.762832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-677d6898-9f7b-4459-9732-3049f3dc47b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:55.809105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:55.842739 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 778/1547] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:56:55.850623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5b218e-bc62-4a45-992f-d862c71268fd None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:55.850930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5b218e-bc62-4a45-992f-d862c71268fd None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:55.851566 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5b218e-bc62-4a45-992f-d862c71268fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:55.893566 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-bf5b218e-bc62-4a45-992f-d862c71268fd None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:55.894713 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-bf5b218e-bc62-4a45-992f-d862c71268fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:55.904054 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 770/1548] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:56:55.907926 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677d6898-9f7b-4459-9732-3049f3dc47b8 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:55.920059 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677d6898-9f7b-4459-9732-3049f3dc47b8 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:55.932679 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e97b3b8-25c7-4ceb-950a-0ffdb875fa51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:55.934151 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e97b3b8-25c7-4ceb-950a-0ffdb875fa51 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:55.934440 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e97b3b8-25c7-4ceb-950a-0ffdb875fa51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:55.934707 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e97b3b8-25c7-4ceb-950a-0ffdb875fa51 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:55.935572 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e97b3b8-25c7-4ceb-950a-0ffdb875fa51 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:56:55.935897 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e97b3b8-25c7-4ceb-950a-0ffdb875fa51 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:55.938226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e97b3b8-25c7-4ceb-950a-0ffdb875fa51 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:55.953611 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 779/1549] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:56:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:56:55.958339 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5b218e-bc62-4a45-992f-d862c71268fd None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:55.969789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5b218e-bc62-4a45-992f-d862c71268fd None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:55.974510 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c7673c-4e13-47a6-88ca-ecbc1793b0f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:55.975230 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c7673c-4e13-47a6-88ca-ecbc1793b0f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:55.975674 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c7673c-4e13-47a6-88ca-ecbc1793b0f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:55.975977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c7673c-4e13-47a6-88ca-ecbc1793b0f5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:55.976634 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c7673c-4e13-47a6-88ca-ecbc1793b0f5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:55.978576 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c7673c-4e13-47a6-88ca-ecbc1793b0f5 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:55.993252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:56:56.008463 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 771/1550] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:56:55 2022] POST /identity/v3/projects => generated 375 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:56:56.012678 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e97b3b8-25c7-4ceb-950a-0ffdb875fa51 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:56.022759 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e97b3b8-25c7-4ceb-950a-0ffdb875fa51 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:56.077395 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f431a0-9900-4a74-8164-b0e4136d9c28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:56.077910 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f431a0-9900-4a74-8164-b0e4136d9c28 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:56.078178 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f431a0-9900-4a74-8164-b0e4136d9c28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:56.078440 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f431a0-9900-4a74-8164-b0e4136d9c28 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:56.079053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f431a0-9900-4a74-8164-b0e4136d9c28 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:56.080997 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f431a0-9900-4a74-8164-b0e4136d9c28 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:56.104380 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 780/1551] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:56:56 2022] POST /identity/v3/users => generated 288 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:56:56.108689 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c7673c-4e13-47a6-88ca-ecbc1793b0f5 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:56.119641 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c7673c-4e13-47a6-88ca-ecbc1793b0f5 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:56.125586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954d625d-01af-440b-b5ed-5db9a01b0a05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:56.126277 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954d625d-01af-440b-b5ed-5db9a01b0a05 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:56.126563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954d625d-01af-440b-b5ed-5db9a01b0a05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:56.126874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954d625d-01af-440b-b5ed-5db9a01b0a05 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:56.127566 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954d625d-01af-440b-b5ed-5db9a01b0a05 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:56.129207 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954d625d-01af-440b-b5ed-5db9a01b0a05 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:56.143896 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 772/1552] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:56 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:56.145103 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f431a0-9900-4a74-8164-b0e4136d9c28 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:56.155963 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f431a0-9900-4a74-8164-b0e4136d9c28 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:56.161026 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6da501-fef6-4bc5-ad8f-6f37d26939a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:56.161669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6da501-fef6-4bc5-ad8f-6f37d26939a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:56.162020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6da501-fef6-4bc5-ad8f-6f37d26939a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:56.162196 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6da501-fef6-4bc5-ad8f-6f37d26939a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:56.163302 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6da501-fef6-4bc5-ad8f-6f37d26939a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:56.164966 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6da501-fef6-4bc5-ad8f-6f37d26939a8 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:56.176088 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 781/1553] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:56:56 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:56:56.180456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954d625d-01af-440b-b5ed-5db9a01b0a05 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:56.195755 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954d625d-01af-440b-b5ed-5db9a01b0a05 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:56.202444 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627f19f6-02e0-4aa6-b45d-c7666b9663c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:56.203223 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627f19f6-02e0-4aa6-b45d-c7666b9663c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:56.203554 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627f19f6-02e0-4aa6-b45d-c7666b9663c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:56.203907 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627f19f6-02e0-4aa6-b45d-c7666b9663c4 admin admin] PATH_INFO: `/v3/projects/edeee3821ad14033b705ab8597d6f339/users/84bb0870b0c344dd8b1e6ecc12150c4e/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:56.220576 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627f19f6-02e0-4aa6-b45d-c7666b9663c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=edeee3821ad14033b705ab8597d6f339, user_id=84bb0870b0c344dd8b1e6ecc12150c4e, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:56.223513 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627f19f6-02e0-4aa6-b45d-c7666b9663c4 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:56.247909 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 773/1554] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:56:56 2022] PUT /identity/v3/projects/edeee3821ad14033b705ab8597d6f339/users/84bb0870b0c344dd8b1e6ecc12150c4e/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:56:56.250494 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143585cf-cb74-4dbd-bf1b-26a462521976 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:56.250772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143585cf-cb74-4dbd-bf1b-26a462521976 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:56.251418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143585cf-cb74-4dbd-bf1b-26a462521976 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:56.280426 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-143585cf-cb74-4dbd-bf1b-26a462521976 None None] MFA Rules not processed for user `84bb0870b0c344dd8b1e6ecc12150c4e`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:56:56.281723 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-143585cf-cb74-4dbd-bf1b-26a462521976 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:56.361249 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 782/1555] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:56:56 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:56:56.371938 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627f19f6-02e0-4aa6-b45d-c7666b9663c4 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:56:56.374106 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-627f19f6-02e0-4aa6-b45d-c7666b9663c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:56:56.472457 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627f19f6-02e0-4aa6-b45d-c7666b9663c4 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:56:56.476866 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d033552-9f45-4b87-981b-fd67d35026dd service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '807b123652d245269336fcffaf36478e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:56:56.477563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d033552-9f45-4b87-981b-fd67d35026dd service glance] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:56:56.477906 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d033552-9f45-4b87-981b-fd67d35026dd service glance] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:56:56.478249 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d033552-9f45-4b87-981b-fd67d35026dd service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:56:56.488782 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d033552-9f45-4b87-981b-fd67d35026dd service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:56:56.490036 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d033552-9f45-4b87-981b-fd67d35026dd service glance] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:56:56.507855 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 774/1556] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:56:56 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:57:02.188747 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143585cf-cb74-4dbd-bf1b-26a462521976 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:57:02.190294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-143585cf-cb74-4dbd-bf1b-26a462521976 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:57:02.230177 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:57:02.254920 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143585cf-cb74-4dbd-bf1b-26a462521976 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:57:02.259158 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21704bc-929d-419a-a88a-751ffb4bac21 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:57:02.259817 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21704bc-929d-419a-a88a-751ffb4bac21 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:57:02.260090 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21704bc-929d-419a-a88a-751ffb4bac21 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:57:02.260350 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21704bc-929d-419a-a88a-751ffb4bac21 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:57:02.262154 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-b21704bc-929d-419a-a88a-751ffb4bac21 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:57:02.330337 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b21704bc-929d-419a-a88a-751ffb4bac21 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:57:02.333086 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b21704bc-929d-419a-a88a-751ffb4bac21 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:57:02.350725 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 783/1557] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:57:02 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:57:06.495918 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:INSERT=2,keystone:SELECT=52 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:57:12.345829 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=35 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:07.940368 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d033552-9f45-4b87-981b-fd67d35026dd service glance] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:07.942195 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-1d033552-9f45-4b87-981b-fd67d35026dd service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:08.012758 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d033552-9f45-4b87-981b-fd67d35026dd service glance] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:08.017676 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd38a669-1d7d-4f07-8962-a10e19147667 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:08.018317 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd38a669-1d7d-4f07-8962-a10e19147667 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:08.018731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd38a669-1d7d-4f07-8962-a10e19147667 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:08.018850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd38a669-1d7d-4f07-8962-a10e19147667 admin admin] PATH_INFO: `/v3/users/ffcee427ba844256a2533311ab7f4963` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:08.019987 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd38a669-1d7d-4f07-8962-a10e19147667 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ffcee427ba844256a2533311ab7f4963)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:08.021892 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd38a669-1d7d-4f07-8962-a10e19147667 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:08.074861 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-dd38a669-1d7d-4f07-8962-a10e19147667 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ffcee427ba844256a2533311ab7f4963'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:08.084112 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-dd38a669-1d7d-4f07-8962-a10e19147667 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ffcee427ba844256a2533311ab7f4963'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:08.084676 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-dd38a669-1d7d-4f07-8962-a10e19147667 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ffcee427ba844256a2533311ab7f4963'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:08.092348 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-dd38a669-1d7d-4f07-8962-a10e19147667 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ffcee427ba844256a2533311ab7f4963'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:08.119322 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-dd38a669-1d7d-4f07-8962-a10e19147667 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ffcee427ba844256a2533311ab7f4963'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:08.127756 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 775/1558] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:58:07 2022] DELETE /identity/v3/users/ffcee427ba844256a2533311ab7f4963 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:08.143963 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784aa548-ec6c-47aa-8a9d-25dd8727fd64 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:08.143963 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784aa548-ec6c-47aa-8a9d-25dd8727fd64 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:08.143963 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784aa548-ec6c-47aa-8a9d-25dd8727fd64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:08.156114 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-784aa548-ec6c-47aa-8a9d-25dd8727fd64 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:08.157525 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-784aa548-ec6c-47aa-8a9d-25dd8727fd64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:08.249879 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 784/1559] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:08.263542 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd38a669-1d7d-4f07-8962-a10e19147667 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:08.265515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-dd38a669-1d7d-4f07-8962-a10e19147667 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:08.345294 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd38a669-1d7d-4f07-8962-a10e19147667 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:08.355102 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd3f20d-a377-4133-8e86-eaca116057a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:08.356144 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd3f20d-a377-4133-8e86-eaca116057a5 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:08.356787 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd3f20d-a377-4133-8e86-eaca116057a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:08.357333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd3f20d-a377-4133-8e86-eaca116057a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:08.372074 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd3f20d-a377-4133-8e86-eaca116057a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:08.373502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd3f20d-a377-4133-8e86-eaca116057a5 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:08.384297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 776/1560] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:08.717985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784aa548-ec6c-47aa-8a9d-25dd8727fd64 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:08.719602 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-784aa548-ec6c-47aa-8a9d-25dd8727fd64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:08.732130 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784aa548-ec6c-47aa-8a9d-25dd8727fd64 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:08.737273 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136bb96a-6742-4452-a18d-461376670b4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:08.738012 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136bb96a-6742-4452-a18d-461376670b4b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:08.738342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136bb96a-6742-4452-a18d-461376670b4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:08.738707 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136bb96a-6742-4452-a18d-461376670b4b admin admin] PATH_INFO: `/v3/projects/5f9254764af44e9dbee0907067094154` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:08.739911 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136bb96a-6742-4452-a18d-461376670b4b admin admin] RBAC: Authorizing `identity:delete_project(project_id=5f9254764af44e9dbee0907067094154)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:08.741920 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136bb96a-6742-4452-a18d-461376670b4b admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:08.789486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-136bb96a-6742-4452-a18d-461376670b4b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5f9254764af44e9dbee0907067094154', 'request_id': 'req-136bb96a-6742-4452-a18d-461376670b4b'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:08.798965 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-136bb96a-6742-4452-a18d-461376670b4b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5f9254764af44e9dbee0907067094154', 'request_id': 'req-136bb96a-6742-4452-a18d-461376670b4b'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:08.807915 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-136bb96a-6742-4452-a18d-461376670b4b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5f9254764af44e9dbee0907067094154', 'request_id': 'req-136bb96a-6742-4452-a18d-461376670b4b'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:08.818150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-136bb96a-6742-4452-a18d-461376670b4b admin admin] The token cache is being invalidate because project 5f9254764af44e9dbee0907067094154 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:58:08.818435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-136bb96a-6742-4452-a18d-461376670b4b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:08.819096 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-136bb96a-6742-4452-a18d-461376670b4b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:08.823924 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 785/1561] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:58:08 2022] DELETE /identity/v3/projects/5f9254764af44e9dbee0907067094154 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:08.825283 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd3f20d-a377-4133-8e86-eaca116057a5 service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:08.835773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd3f20d-a377-4133-8e86-eaca116057a5 service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:08.857945 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:08.903373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b439f0-f5eb-426c-acc0-da1956557d58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:08.904060 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b439f0-f5eb-426c-acc0-da1956557d58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:08.904337 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b439f0-f5eb-426c-acc0-da1956557d58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:08.904604 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b439f0-f5eb-426c-acc0-da1956557d58 admin admin] PATH_INFO: `/v3/users/329fbcbc1e824a6aa5fe4f4eab099243` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:08.905853 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b439f0-f5eb-426c-acc0-da1956557d58 admin admin] RBAC: Authorizing `identity:delete_user(user_id=329fbcbc1e824a6aa5fe4f4eab099243)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:08.908206 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b439f0-f5eb-426c-acc0-da1956557d58 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:08.957251 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a2b439f0-f5eb-426c-acc0-da1956557d58 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '329fbcbc1e824a6aa5fe4f4eab099243'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:08.965340 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a2b439f0-f5eb-426c-acc0-da1956557d58 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '329fbcbc1e824a6aa5fe4f4eab099243'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:08.966063 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a2b439f0-f5eb-426c-acc0-da1956557d58 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '329fbcbc1e824a6aa5fe4f4eab099243'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:08.973525 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a2b439f0-f5eb-426c-acc0-da1956557d58 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '329fbcbc1e824a6aa5fe4f4eab099243'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:08.985632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a2b439f0-f5eb-426c-acc0-da1956557d58 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '329fbcbc1e824a6aa5fe4f4eab099243'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:08.992106 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 777/1562] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:58:08 2022] DELETE /identity/v3/users/329fbcbc1e824a6aa5fe4f4eab099243 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:09.484065 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136bb96a-6742-4452-a18d-461376670b4b admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:09.486464 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-136bb96a-6742-4452-a18d-461376670b4b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:09.503184 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136bb96a-6742-4452-a18d-461376670b4b admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:09.565571 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7713a4-4767-4447-acae-7926d9d62686 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:09.566214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7713a4-4767-4447-acae-7926d9d62686 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:09.566479 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7713a4-4767-4447-acae-7926d9d62686 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:09.566724 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7713a4-4767-4447-acae-7926d9d62686 admin admin] PATH_INFO: `/v3/users/cebabbfa35d84f28b02f3d46797a3129` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:09.567903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7713a4-4767-4447-acae-7926d9d62686 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cebabbfa35d84f28b02f3d46797a3129)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:09.569919 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7713a4-4767-4447-acae-7926d9d62686 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:09.600455 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b439f0-f5eb-426c-acc0-da1956557d58 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:09.603031 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a2b439f0-f5eb-426c-acc0-da1956557d58 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:09.618441 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:09.623159 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b439f0-f5eb-426c-acc0-da1956557d58 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:09.630219 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970c4528-02c3-4d79-8ef3-095768ad05b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:09.631028 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970c4528-02c3-4d79-8ef3-095768ad05b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:09.631432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970c4528-02c3-4d79-8ef3-095768ad05b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:09.631775 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970c4528-02c3-4d79-8ef3-095768ad05b9 admin admin] PATH_INFO: `/v3/users/84bb0870b0c344dd8b1e6ecc12150c4e` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:09.633018 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970c4528-02c3-4d79-8ef3-095768ad05b9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=84bb0870b0c344dd8b1e6ecc12150c4e)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:09.634999 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970c4528-02c3-4d79-8ef3-095768ad05b9 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:09.647442 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ca7713a4-4767-4447-acae-7926d9d62686 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cebabbfa35d84f28b02f3d46797a3129'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:09.657400 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ca7713a4-4767-4447-acae-7926d9d62686 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cebabbfa35d84f28b02f3d46797a3129'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:09.658219 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ca7713a4-4767-4447-acae-7926d9d62686 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cebabbfa35d84f28b02f3d46797a3129'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:09.681904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ca7713a4-4767-4447-acae-7926d9d62686 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cebabbfa35d84f28b02f3d46797a3129'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:09.691128 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ca7713a4-4767-4447-acae-7926d9d62686 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cebabbfa35d84f28b02f3d46797a3129'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:09.692623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-970c4528-02c3-4d79-8ef3-095768ad05b9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '84bb0870b0c344dd8b1e6ecc12150c4e'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:09.703074 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-970c4528-02c3-4d79-8ef3-095768ad05b9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '84bb0870b0c344dd8b1e6ecc12150c4e'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:09.704701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-970c4528-02c3-4d79-8ef3-095768ad05b9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '84bb0870b0c344dd8b1e6ecc12150c4e'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:09.707192 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 786/1563] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:58:09 2022] DELETE /identity/v3/users/cebabbfa35d84f28b02f3d46797a3129 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:09.712323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c1b3a0-f514-49ef-9248-cc6b7340b7fe None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:09.712603 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c1b3a0-f514-49ef-9248-cc6b7340b7fe None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:09.712872 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c1b3a0-f514-49ef-9248-cc6b7340b7fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:09.721347 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-970c4528-02c3-4d79-8ef3-095768ad05b9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '84bb0870b0c344dd8b1e6ecc12150c4e'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:09.737382 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-970c4528-02c3-4d79-8ef3-095768ad05b9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '84bb0870b0c344dd8b1e6ecc12150c4e'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:09.742629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-b9c1b3a0-f514-49ef-9248-cc6b7340b7fe None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:09.744223 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 778/1564] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:58:09 2022] DELETE /identity/v3/users/84bb0870b0c344dd8b1e6ecc12150c4e => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:09.745181 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970c4528-02c3-4d79-8ef3-095768ad05b9 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:09.746637 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-970c4528-02c3-4d79-8ef3-095768ad05b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:09.749830 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-b9c1b3a0-f514-49ef-9248-cc6b7340b7fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:09.769305 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:09.859476 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 787/1565] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:09.861195 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782165ee-cc88-4cb5-92c0-88cff6d78814 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:09.861722 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782165ee-cc88-4cb5-92c0-88cff6d78814 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:09.862087 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782165ee-cc88-4cb5-92c0-88cff6d78814 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:09.869429 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970c4528-02c3-4d79-8ef3-095768ad05b9 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:09.878125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1f2b38-bd65-444b-91ae-1ab521a9d257 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:09.879241 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1f2b38-bd65-444b-91ae-1ab521a9d257 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:09.879526 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1f2b38-bd65-444b-91ae-1ab521a9d257 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:09.879946 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1f2b38-bd65-444b-91ae-1ab521a9d257 admin admin] PATH_INFO: `/v3/projects/8435251b59f742dfa7c4366d91a7c10d` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:09.881780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1f2b38-bd65-444b-91ae-1ab521a9d257 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8435251b59f742dfa7c4366d91a7c10d)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:09.885370 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1f2b38-bd65-444b-91ae-1ab521a9d257 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:09.888668 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-782165ee-cc88-4cb5-92c0-88cff6d78814 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:09.890498 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-782165ee-cc88-4cb5-92c0-88cff6d78814 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:09.899912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 788/1566] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:09.901001 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782165ee-cc88-4cb5-92c0-88cff6d78814 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:09.902686 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-782165ee-cc88-4cb5-92c0-88cff6d78814 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:09.956402 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-0f1f2b38-bd65-444b-91ae-1ab521a9d257 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8435251b59f742dfa7c4366d91a7c10d', 'request_id': 'req-0f1f2b38-bd65-444b-91ae-1ab521a9d257'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:09.962686 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-0f1f2b38-bd65-444b-91ae-1ab521a9d257 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8435251b59f742dfa7c4366d91a7c10d', 'request_id': 'req-0f1f2b38-bd65-444b-91ae-1ab521a9d257'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:09.974476 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-0f1f2b38-bd65-444b-91ae-1ab521a9d257 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8435251b59f742dfa7c4366d91a7c10d', 'request_id': 'req-0f1f2b38-bd65-444b-91ae-1ab521a9d257'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:09.986081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782165ee-cc88-4cb5-92c0-88cff6d78814 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:09.986921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-0f1f2b38-bd65-444b-91ae-1ab521a9d257 admin admin] The token cache is being invalidate because project 8435251b59f742dfa7c4366d91a7c10d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:58:09.987300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-0f1f2b38-bd65-444b-91ae-1ab521a9d257 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:09.988027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-0f1f2b38-bd65-444b-91ae-1ab521a9d257 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:09.989892 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 779/1567] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:58:09 2022] DELETE /identity/v3/projects/8435251b59f742dfa7c4366d91a7c10d => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:09.990659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1f2b38-bd65-444b-91ae-1ab521a9d257 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:09.991953 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-0f1f2b38-bd65-444b-91ae-1ab521a9d257 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:10.056198 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34be649c-99fc-4b3b-85df-febd4629e2dc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:10.057596 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34be649c-99fc-4b3b-85df-febd4629e2dc service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:10.057875 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34be649c-99fc-4b3b-85df-febd4629e2dc service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:10.058142 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34be649c-99fc-4b3b-85df-febd4629e2dc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:10.060203 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-34be649c-99fc-4b3b-85df-febd4629e2dc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:10.060916 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1f2b38-bd65-444b-91ae-1ab521a9d257 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:10.064624 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a9f07e-2073-46fa-8035-fe8841442628 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:10.065238 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a9f07e-2073-46fa-8035-fe8841442628 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:10.065509 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a9f07e-2073-46fa-8035-fe8841442628 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:10.065768 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a9f07e-2073-46fa-8035-fe8841442628 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:10.066538 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:DELETE=6,keystone:SELECT=93,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:10.067480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-62a9f07e-2073-46fa-8035-fe8841442628 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:10.132484 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34be649c-99fc-4b3b-85df-febd4629e2dc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:10.134459 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34be649c-99fc-4b3b-85df-febd4629e2dc service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:10.138242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:10.146474 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a9f07e-2073-46fa-8035-fe8841442628 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:10.147581 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 789/1568] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 247 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:10.148561 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a9f07e-2073-46fa-8035-fe8841442628 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:10.149759 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d634f59-ceca-409a-9c93-2cd1d2376392 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:10.150587 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d634f59-ceca-409a-9c93-2cd1d2376392 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:10.150870 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d634f59-ceca-409a-9c93-2cd1d2376392 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:10.164451 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 780/1569] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:10.176039 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-3d634f59-ceca-409a-9c93-2cd1d2376392 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:10.177309 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3d634f59-ceca-409a-9c93-2cd1d2376392 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:10.187830 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 790/1570] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:10.196902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b247d4-a3c3-4f4c-9d47-bd4e533c7b17 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:10.197691 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b247d4-a3c3-4f4c-9d47-bd4e533c7b17 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:10.197964 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b247d4-a3c3-4f4c-9d47-bd4e533c7b17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:10.220385 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-11b247d4-a3c3-4f4c-9d47-bd4e533c7b17 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:10.221680 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-11b247d4-a3c3-4f4c-9d47-bd4e533c7b17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:10.230727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 781/1571] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:10.235540 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d634f59-ceca-409a-9c93-2cd1d2376392 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:10.248043 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d634f59-ceca-409a-9c93-2cd1d2376392 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:10.253440 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f027e049-6f29-40a7-bf85-db1844016f11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:10.254200 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f027e049-6f29-40a7-bf85-db1844016f11 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:10.254316 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f027e049-6f29-40a7-bf85-db1844016f11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:10.254910 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f027e049-6f29-40a7-bf85-db1844016f11 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:10.255605 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f027e049-6f29-40a7-bf85-db1844016f11 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:58:10.256282 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f027e049-6f29-40a7-bf85-db1844016f11 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:10.258322 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f027e049-6f29-40a7-bf85-db1844016f11 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:10.276784 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 791/1572] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:58:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:58:10.278384 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b247d4-a3c3-4f4c-9d47-bd4e533c7b17 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:10.290276 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b247d4-a3c3-4f4c-9d47-bd4e533c7b17 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:10.296996 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd504a4d-e75f-4bef-9cf0-5f34a46b5a25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:10.297623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd504a4d-e75f-4bef-9cf0-5f34a46b5a25 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:10.297895 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd504a4d-e75f-4bef-9cf0-5f34a46b5a25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:10.298206 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd504a4d-e75f-4bef-9cf0-5f34a46b5a25 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:10.298842 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd504a4d-e75f-4bef-9cf0-5f34a46b5a25 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:10.301574 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd504a4d-e75f-4bef-9cf0-5f34a46b5a25 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:10.345892 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 782/1573] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:58:10 2022] POST /identity/v3/projects => generated 393 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:58:10.348208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f027e049-6f29-40a7-bf85-db1844016f11 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:10.358914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f027e049-6f29-40a7-bf85-db1844016f11 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:10.450813 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9c6d43-2ec9-4dac-98d7-6332f695311b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:10.451438 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9c6d43-2ec9-4dac-98d7-6332f695311b admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:10.451730 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9c6d43-2ec9-4dac-98d7-6332f695311b admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:10.452023 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9c6d43-2ec9-4dac-98d7-6332f695311b admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:10.452716 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9c6d43-2ec9-4dac-98d7-6332f695311b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:10.454769 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9c6d43-2ec9-4dac-98d7-6332f695311b admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:10.504067 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 792/1574] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:58:10 2022] POST /identity/v3/users => generated 297 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:10.508454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd504a4d-e75f-4bef-9cf0-5f34a46b5a25 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:10.524928 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd504a4d-e75f-4bef-9cf0-5f34a46b5a25 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:10.530373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de2d14f-71fd-4399-89b1-87635ea9b111 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:10.531169 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de2d14f-71fd-4399-89b1-87635ea9b111 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:10.531542 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de2d14f-71fd-4399-89b1-87635ea9b111 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:10.531968 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de2d14f-71fd-4399-89b1-87635ea9b111 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:10.532716 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de2d14f-71fd-4399-89b1-87635ea9b111 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:10.535640 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de2d14f-71fd-4399-89b1-87635ea9b111 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:10.551057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 783/1575] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:10 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:10.555119 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9c6d43-2ec9-4dac-98d7-6332f695311b admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:10.566749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9c6d43-2ec9-4dac-98d7-6332f695311b admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:10.596628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc18b6d2-38e1-4d75-87af-d06cf30cc5b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:10.598814 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc18b6d2-38e1-4d75-87af-d06cf30cc5b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:10.599318 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc18b6d2-38e1-4d75-87af-d06cf30cc5b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:10.599701 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc18b6d2-38e1-4d75-87af-d06cf30cc5b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:10.600479 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc18b6d2-38e1-4d75-87af-d06cf30cc5b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:10.602887 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc18b6d2-38e1-4d75-87af-d06cf30cc5b6 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:10.616071 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 793/1576] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:10 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:10.619986 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de2d14f-71fd-4399-89b1-87635ea9b111 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:10.631633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de2d14f-71fd-4399-89b1-87635ea9b111 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:10.638705 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b48d2b-6efd-438b-a194-40945fc0fb74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:10.639548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b48d2b-6efd-438b-a194-40945fc0fb74 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:10.639850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b48d2b-6efd-438b-a194-40945fc0fb74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:10.640200 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b48d2b-6efd-438b-a194-40945fc0fb74 admin admin] PATH_INFO: `/v3/projects/087b66d74f7a4b67820b8d0872eaa280/users/44be814d35bc49888ef6aa629f6af604/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:10.663211 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b48d2b-6efd-438b-a194-40945fc0fb74 admin admin] RBAC: Authorizing `identity:create_grant(project_id=087b66d74f7a4b67820b8d0872eaa280, user_id=44be814d35bc49888ef6aa629f6af604, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:10.667805 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b48d2b-6efd-438b-a194-40945fc0fb74 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:10.684425 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc18b6d2-38e1-4d75-87af-d06cf30cc5b6 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:10.686554 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-dc18b6d2-38e1-4d75-87af-d06cf30cc5b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:10.686793 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 784/1577] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:10 2022] PUT /identity/v3/projects/087b66d74f7a4b67820b8d0872eaa280/users/44be814d35bc49888ef6aa629f6af604/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:10.699156 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4644bd22-70f5-4fe8-96bb-d52890d93e0e None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:10.699443 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4644bd22-70f5-4fe8-96bb-d52890d93e0e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:10.699708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4644bd22-70f5-4fe8-96bb-d52890d93e0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:10.715470 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc18b6d2-38e1-4d75-87af-d06cf30cc5b6 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:10.737120 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-4644bd22-70f5-4fe8-96bb-d52890d93e0e None None] MFA Rules not processed for user `44be814d35bc49888ef6aa629f6af604`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:10.739017 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-4644bd22-70f5-4fe8-96bb-d52890d93e0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:10.820564 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4b6051-7649-40ac-a181-708014243d01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:10.822437 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4b6051-7649-40ac-a181-708014243d01 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:10.822782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4b6051-7649-40ac-a181-708014243d01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:10.823127 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4b6051-7649-40ac-a181-708014243d01 admin admin] PATH_INFO: `/v3/projects/e9c26f5f988f415693f0d561736976d2` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:10.824429 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4b6051-7649-40ac-a181-708014243d01 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e9c26f5f988f415693f0d561736976d2)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:10.826879 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4b6051-7649-40ac-a181-708014243d01 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:10.831272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:10.832614 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 785/1578] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:10 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:10.838704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4644bd22-70f5-4fe8-96bb-d52890d93e0e None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:10.852579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4644bd22-70f5-4fe8-96bb-d52890d93e0e None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:10.858113 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0defe2-dd2f-453f-9a6f-03bf7e27e057 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:10.858810 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0defe2-dd2f-453f-9a6f-03bf7e27e057 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:10.859090 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0defe2-dd2f-453f-9a6f-03bf7e27e057 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:10.859505 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0defe2-dd2f-453f-9a6f-03bf7e27e057 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:10.860209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0defe2-dd2f-453f-9a6f-03bf7e27e057 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:10.863077 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0defe2-dd2f-453f-9a6f-03bf7e27e057 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:10.914522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 786/1579] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:58:10 2022] POST /identity/v3/projects => generated 393 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:58:10.915180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0defe2-dd2f-453f-9a6f-03bf7e27e057 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:10.920501 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-8b0defe2-dd2f-453f-9a6f-03bf7e27e057 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:10.921747 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ff4b6051-7649-40ac-a181-708014243d01 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e9c26f5f988f415693f0d561736976d2', 'request_id': 'req-ff4b6051-7649-40ac-a181-708014243d01'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:10.933861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ff4b6051-7649-40ac-a181-708014243d01 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e9c26f5f988f415693f0d561736976d2', 'request_id': 'req-ff4b6051-7649-40ac-a181-708014243d01'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:10.944514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ff4b6051-7649-40ac-a181-708014243d01 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e9c26f5f988f415693f0d561736976d2', 'request_id': 'req-ff4b6051-7649-40ac-a181-708014243d01'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:10.992616 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ff4b6051-7649-40ac-a181-708014243d01 admin admin] The token cache is being invalidate because project e9c26f5f988f415693f0d561736976d2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:58:10.992930 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ff4b6051-7649-40ac-a181-708014243d01 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:10.995243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ff4b6051-7649-40ac-a181-708014243d01 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:10.998612 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 794/1580] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:58:10 2022] DELETE /identity/v3/projects/e9c26f5f988f415693f0d561736976d2 => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:11.001708 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4b6051-7649-40ac-a181-708014243d01 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:11.004164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ff4b6051-7649-40ac-a181-708014243d01 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:11.004738 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:11.012531 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0defe2-dd2f-453f-9a6f-03bf7e27e057 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:11.028340 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023e1e61-a4e7-48ca-a722-78f55e6f1fe3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:11.029620 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023e1e61-a4e7-48ca-a722-78f55e6f1fe3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:11.029997 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023e1e61-a4e7-48ca-a722-78f55e6f1fe3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:11.030370 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023e1e61-a4e7-48ca-a722-78f55e6f1fe3 admin admin] PATH_INFO: `/v3/projects/edeee3821ad14033b705ab8597d6f339` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:11.032485 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023e1e61-a4e7-48ca-a722-78f55e6f1fe3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=edeee3821ad14033b705ab8597d6f339)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:11.034527 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023e1e61-a4e7-48ca-a722-78f55e6f1fe3 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:11.120100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4b6051-7649-40ac-a181-708014243d01 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:11.126168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-023e1e61-a4e7-48ca-a722-78f55e6f1fe3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'edeee3821ad14033b705ab8597d6f339', 'request_id': 'req-023e1e61-a4e7-48ca-a722-78f55e6f1fe3'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:11.136862 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-023e1e61-a4e7-48ca-a722-78f55e6f1fe3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'edeee3821ad14033b705ab8597d6f339', 'request_id': 'req-023e1e61-a4e7-48ca-a722-78f55e6f1fe3'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:11.159116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-023e1e61-a4e7-48ca-a722-78f55e6f1fe3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'edeee3821ad14033b705ab8597d6f339', 'request_id': 'req-023e1e61-a4e7-48ca-a722-78f55e6f1fe3'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:11.182319 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-023e1e61-a4e7-48ca-a722-78f55e6f1fe3 admin admin] The token cache is being invalidate because project edeee3821ad14033b705ab8597d6f339 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:58:11.182809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-023e1e61-a4e7-48ca-a722-78f55e6f1fe3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:11.184439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-023e1e61-a4e7-48ca-a722-78f55e6f1fe3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:11.189635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 787/1581] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:58:10 2022] DELETE /identity/v3/projects/edeee3821ad14033b705ab8597d6f339 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:11.194174 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023e1e61-a4e7-48ca-a722-78f55e6f1fe3 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:11.198196 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-023e1e61-a4e7-48ca-a722-78f55e6f1fe3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:11.211763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff18c641-8674-419f-b864-cf8a59767d14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:11.211763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff18c641-8674-419f-b864-cf8a59767d14 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:11.211763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff18c641-8674-419f-b864-cf8a59767d14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:11.211763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff18c641-8674-419f-b864-cf8a59767d14 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:11.212282 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff18c641-8674-419f-b864-cf8a59767d14 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:11.212282 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff18c641-8674-419f-b864-cf8a59767d14 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:11.223206 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023e1e61-a4e7-48ca-a722-78f55e6f1fe3 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:11.237529 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbaaeb48-26d9-4d3e-9085-a7d5d6e62471 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:11.239326 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbaaeb48-26d9-4d3e-9085-a7d5d6e62471 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:11.239621 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbaaeb48-26d9-4d3e-9085-a7d5d6e62471 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:11.239926 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbaaeb48-26d9-4d3e-9085-a7d5d6e62471 admin admin] PATH_INFO: `/v3/users/e0bc45577a024bcdb4fd97faa2c988ad` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:11.244759 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbaaeb48-26d9-4d3e-9085-a7d5d6e62471 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e0bc45577a024bcdb4fd97faa2c988ad)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:11.248592 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbaaeb48-26d9-4d3e-9085-a7d5d6e62471 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:11.249471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 795/1582] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:58:10 2022] POST /identity/v3/users => generated 303 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:11.252133 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94d7b9d-c4da-423d-b4de-2ed8d8852991 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:11.252743 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94d7b9d-c4da-423d-b4de-2ed8d8852991 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:11.253806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94d7b9d-c4da-423d-b4de-2ed8d8852991 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:11.280692 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-e94d7b9d-c4da-423d-b4de-2ed8d8852991 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:11.282424 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e94d7b9d-c4da-423d-b4de-2ed8d8852991 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:11.295473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 796/1583] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:11.297011 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94d7b9d-c4da-423d-b4de-2ed8d8852991 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:11.299399 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e94d7b9d-c4da-423d-b4de-2ed8d8852991 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:11.320594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94d7b9d-c4da-423d-b4de-2ed8d8852991 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:11.328736 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c112d977-1e13-46bf-86b9-68be6e6e9912 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:11.329809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c112d977-1e13-46bf-86b9-68be6e6e9912 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:11.330652 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c112d977-1e13-46bf-86b9-68be6e6e9912 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:11.331091 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c112d977-1e13-46bf-86b9-68be6e6e9912 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:11.332725 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c112d977-1e13-46bf-86b9-68be6e6e9912 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:11.336891 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c112d977-1e13-46bf-86b9-68be6e6e9912 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:11.358335 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 797/1584] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:11 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:11.371483 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-bbaaeb48-26d9-4d3e-9085-a7d5d6e62471 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e0bc45577a024bcdb4fd97faa2c988ad'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:11.373042 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365e8ff-79f6-49a7-9ce7-7a47f54dec5d None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:11.373432 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365e8ff-79f6-49a7-9ce7-7a47f54dec5d None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:11.373711 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365e8ff-79f6-49a7-9ce7-7a47f54dec5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:11.385824 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-bbaaeb48-26d9-4d3e-9085-a7d5d6e62471 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e0bc45577a024bcdb4fd97faa2c988ad'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:11.386726 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-bbaaeb48-26d9-4d3e-9085-a7d5d6e62471 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e0bc45577a024bcdb4fd97faa2c988ad'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:11.395527 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-bbaaeb48-26d9-4d3e-9085-a7d5d6e62471 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e0bc45577a024bcdb4fd97faa2c988ad'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:11.401956 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-f365e8ff-79f6-49a7-9ce7-7a47f54dec5d None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:11.404087 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-f365e8ff-79f6-49a7-9ce7-7a47f54dec5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:11.413308 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-bbaaeb48-26d9-4d3e-9085-a7d5d6e62471 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e0bc45577a024bcdb4fd97faa2c988ad'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:11.417355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 798/1585] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:11.419192 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f365e8ff-79f6-49a7-9ce7-7a47f54dec5d None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:11.421076 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-f365e8ff-79f6-49a7-9ce7-7a47f54dec5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:11.428401 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 788/1586] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:58:11 2022] DELETE /identity/v3/users/e0bc45577a024bcdb4fd97faa2c988ad => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:11.432541 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:11.432745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbaaeb48-26d9-4d3e-9085-a7d5d6e62471 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:11.437469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f365e8ff-79f6-49a7-9ce7-7a47f54dec5d None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:11.564057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbaaeb48-26d9-4d3e-9085-a7d5d6e62471 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:11.570508 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34704b2-72ea-4b1f-8c69-7ea08ee4d9c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:11.571244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34704b2-72ea-4b1f-8c69-7ea08ee4d9c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:11.571516 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34704b2-72ea-4b1f-8c69-7ea08ee4d9c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:11.571967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34704b2-72ea-4b1f-8c69-7ea08ee4d9c4 admin admin] PATH_INFO: `/v3/projects/3dff670d1b1d43b2a3e6a9f41b4e25d5/users/88e655dc848e429c9babb23643116bde/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:11.579806 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb0b9f5-1edf-4d24-873d-6c9c0849c655 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:11.580592 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb0b9f5-1edf-4d24-873d-6c9c0849c655 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:11.580964 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb0b9f5-1edf-4d24-873d-6c9c0849c655 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:11.581328 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb0b9f5-1edf-4d24-873d-6c9c0849c655 admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:11.582093 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb0b9f5-1edf-4d24-873d-6c9c0849c655 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:58:11.582972 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb0b9f5-1edf-4d24-873d-6c9c0849c655 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:11.587697 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb0b9f5-1edf-4d24-873d-6c9c0849c655 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:11.592903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:11.595505 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d34704b2-72ea-4b1f-8c69-7ea08ee4d9c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3dff670d1b1d43b2a3e6a9f41b4e25d5, user_id=88e655dc848e429c9babb23643116bde, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:11.601976 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d34704b2-72ea-4b1f-8c69-7ea08ee4d9c4 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:11.621354 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 799/1587] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:11 2022] PUT /identity/v3/projects/3dff670d1b1d43b2a3e6a9f41b4e25d5/users/88e655dc848e429c9babb23643116bde/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:11.627192 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34704b2-72ea-4b1f-8c69-7ea08ee4d9c4 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:11.634545 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 789/1588] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:58:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:58:11.636825 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb0b9f5-1edf-4d24-873d-6c9c0849c655 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:11.673238 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb0b9f5-1edf-4d24-873d-6c9c0849c655 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:11.753343 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34704b2-72ea-4b1f-8c69-7ea08ee4d9c4 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:11.764124 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b9c495-7f24-4128-a350-eefdec5bfa0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:11.764889 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b9c495-7f24-4128-a350-eefdec5bfa0e admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:11.765275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b9c495-7f24-4128-a350-eefdec5bfa0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:11.765634 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b9c495-7f24-4128-a350-eefdec5bfa0e admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:11.766419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b9c495-7f24-4128-a350-eefdec5bfa0e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:11.772162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b9c495-7f24-4128-a350-eefdec5bfa0e admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:11.797739 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28eafbc-e38e-4c91-9b8d-0314dc3c57f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:11.798756 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28eafbc-e38e-4c91-9b8d-0314dc3c57f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:11.799272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28eafbc-e38e-4c91-9b8d-0314dc3c57f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:11.800005 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28eafbc-e38e-4c91-9b8d-0314dc3c57f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:11.800991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c28eafbc-e38e-4c91-9b8d-0314dc3c57f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:11.803103 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c28eafbc-e38e-4c91-9b8d-0314dc3c57f1 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:11.814181 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 800/1589] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:58:11 2022] POST /identity/v3/projects => generated 375 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:11.815395 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b9c495-7f24-4128-a350-eefdec5bfa0e admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:11.820504 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 790/1590] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:11 2022] GET /identity/v3/roles => generated 1420 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:11.827553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28eafbc-e38e-4c91-9b8d-0314dc3c57f1 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:11.872080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28eafbc-e38e-4c91-9b8d-0314dc3c57f1 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:11.951281 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b9c495-7f24-4128-a350-eefdec5bfa0e admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:11.964748 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b161b940-97e9-491e-8ff9-8df63b8a9c54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:11.965555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b161b940-97e9-491e-8ff9-8df63b8a9c54 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:11.966683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b161b940-97e9-491e-8ff9-8df63b8a9c54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:11.967205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b161b940-97e9-491e-8ff9-8df63b8a9c54 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:11.968002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b161b940-97e9-491e-8ff9-8df63b8a9c54 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:11.974348 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b161b940-97e9-491e-8ff9-8df63b8a9c54 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:12.026847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b560cfa3-ccc6-45ab-8901-1fd4e2dff182 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:12.028476 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 801/1591] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:58:11 2022] POST /identity/v3/users => generated 288 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:12.028599 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b560cfa3-ccc6-45ab-8901-1fd4e2dff182 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:12.028842 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b560cfa3-ccc6-45ab-8901-1fd4e2dff182 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:12.029393 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b560cfa3-ccc6-45ab-8901-1fd4e2dff182 admin admin] PATH_INFO: `/v3/projects/3dff670d1b1d43b2a3e6a9f41b4e25d5/users/88e655dc848e429c9babb23643116bde/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:12.033461 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b161b940-97e9-491e-8ff9-8df63b8a9c54 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:12.034696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b560cfa3-ccc6-45ab-8901-1fd4e2dff182 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3dff670d1b1d43b2a3e6a9f41b4e25d5, user_id=88e655dc848e429c9babb23643116bde, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:12.038088 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b560cfa3-ccc6-45ab-8901-1fd4e2dff182 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:12.050226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:12.054658 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b161b940-97e9-491e-8ff9-8df63b8a9c54 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:12.075379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 791/1592] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:11 2022] PUT /identity/v3/projects/3dff670d1b1d43b2a3e6a9f41b4e25d5/users/88e655dc848e429c9babb23643116bde/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:12.087369 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b560cfa3-ccc6-45ab-8901-1fd4e2dff182 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:12.090676 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b560cfa3-ccc6-45ab-8901-1fd4e2dff182 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:12.162764 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbf4c38-15c3-4a6c-9b6d-45e5f7771108 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:12.163498 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbf4c38-15c3-4a6c-9b6d-45e5f7771108 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:12.163918 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbf4c38-15c3-4a6c-9b6d-45e5f7771108 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:12.164197 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbf4c38-15c3-4a6c-9b6d-45e5f7771108 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:12.165179 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbf4c38-15c3-4a6c-9b6d-45e5f7771108 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:12.167510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbf4c38-15c3-4a6c-9b6d-45e5f7771108 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:12.182090 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 802/1593] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:12 2022] GET /identity/v3/roles => generated 1420 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 00:58:12.185916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03058fde-eb29-4255-8de2-1843b700dcac None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:12.186487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03058fde-eb29-4255-8de2-1843b700dcac None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:12.187119 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03058fde-eb29-4255-8de2-1843b700dcac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:12.198853 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b560cfa3-ccc6-45ab-8901-1fd4e2dff182 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:12.204308 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b08088f-93c7-4c4f-8da9-43c7c5000ecd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:12.205133 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b08088f-93c7-4c4f-8da9-43c7c5000ecd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:12.205513 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b08088f-93c7-4c4f-8da9-43c7c5000ecd admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:12.205956 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b08088f-93c7-4c4f-8da9-43c7c5000ecd admin admin] PATH_INFO: `/v3/projects/ea216333fa5c45c79f656ee70fad8b8b` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:12.207463 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b08088f-93c7-4c4f-8da9-43c7c5000ecd admin admin] RBAC: Authorizing `identity:delete_project(project_id=ea216333fa5c45c79f656ee70fad8b8b)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:12.209428 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b08088f-93c7-4c4f-8da9-43c7c5000ecd admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:12.228925 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-03058fde-eb29-4255-8de2-1843b700dcac None None] MFA Rules not processed for user `88e655dc848e429c9babb23643116bde`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:12.242094 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-03058fde-eb29-4255-8de2-1843b700dcac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:12.246077 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:12.279764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-9b08088f-93c7-4c4f-8da9-43c7c5000ecd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ea216333fa5c45c79f656ee70fad8b8b', 'request_id': 'req-9b08088f-93c7-4c4f-8da9-43c7c5000ecd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:12.286821 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-9b08088f-93c7-4c4f-8da9-43c7c5000ecd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ea216333fa5c45c79f656ee70fad8b8b', 'request_id': 'req-9b08088f-93c7-4c4f-8da9-43c7c5000ecd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:12.298059 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-9b08088f-93c7-4c4f-8da9-43c7c5000ecd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ea216333fa5c45c79f656ee70fad8b8b', 'request_id': 'req-9b08088f-93c7-4c4f-8da9-43c7c5000ecd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:12.320675 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-9b08088f-93c7-4c4f-8da9-43c7c5000ecd admin admin] The token cache is being invalidate because project ea216333fa5c45c79f656ee70fad8b8b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:58:12.321058 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-9b08088f-93c7-4c4f-8da9-43c7c5000ecd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:12.321854 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-9b08088f-93c7-4c4f-8da9-43c7c5000ecd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:12.324349 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 792/1594] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:58:12 2022] DELETE /identity/v3/projects/ea216333fa5c45c79f656ee70fad8b8b => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:12.324991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b08088f-93c7-4c4f-8da9-43c7c5000ecd admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:12.326637 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-9b08088f-93c7-4c4f-8da9-43c7c5000ecd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:12.397075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 803/1595] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:12 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:12.400867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129071e0-3be5-46ad-86fa-ece6d84ef55b None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:12.401318 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129071e0-3be5-46ad-86fa-ece6d84ef55b None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:12.401774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129071e0-3be5-46ad-86fa-ece6d84ef55b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:12.438993 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-129071e0-3be5-46ad-86fa-ece6d84ef55b None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:12.440231 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-129071e0-3be5-46ad-86fa-ece6d84ef55b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:12.447560 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b08088f-93c7-4c4f-8da9-43c7c5000ecd admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:12.449530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 804/1596] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:12.450200 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129071e0-3be5-46ad-86fa-ece6d84ef55b None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:12.451635 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-129071e0-3be5-46ad-86fa-ece6d84ef55b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:12.454015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc873a25-8c09-463e-8b1f-98afb089c763 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:12.454638 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc873a25-8c09-463e-8b1f-98afb089c763 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:12.454914 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc873a25-8c09-463e-8b1f-98afb089c763 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:12.455181 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc873a25-8c09-463e-8b1f-98afb089c763 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:12.455858 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc873a25-8c09-463e-8b1f-98afb089c763 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:12.457472 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc873a25-8c09-463e-8b1f-98afb089c763 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:12.471987 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 793/1597] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:12 2022] GET /identity/v3/roles => generated 1420 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:12.473715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9219c06-d76d-48d9-a821-a04025bd471b None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:12.474101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9219c06-d76d-48d9-a821-a04025bd471b None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:12.474619 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9219c06-d76d-48d9-a821-a04025bd471b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:12.498765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-c9219c06-d76d-48d9-a821-a04025bd471b None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:12.500263 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-c9219c06-d76d-48d9-a821-a04025bd471b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:12.516094 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 794/1598] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:12.517411 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9219c06-d76d-48d9-a821-a04025bd471b None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:12.529341 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129071e0-3be5-46ad-86fa-ece6d84ef55b None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:12.530199 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9219c06-d76d-48d9-a821-a04025bd471b None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:12.533427 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec60623-7747-4990-9b3f-8f1dcb000dbb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:12.534585 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec60623-7747-4990-9b3f-8f1dcb000dbb service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:12.534872 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec60623-7747-4990-9b3f-8f1dcb000dbb service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:12.535228 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec60623-7747-4990-9b3f-8f1dcb000dbb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:12.535699 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d92910-7705-4925-9b43-585793a91cd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:12.536595 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d92910-7705-4925-9b43-585793a91cd1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:12.536965 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d92910-7705-4925-9b43-585793a91cd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:12.537344 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d92910-7705-4925-9b43-585793a91cd1 admin admin] PATH_INFO: `/v3/projects/4370999c56184197856c3b8b25fd2e20/users/0dfb161b6a9144f9aa0d4cf47d90ab20/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:12.556999 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d92910-7705-4925-9b43-585793a91cd1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4370999c56184197856c3b8b25fd2e20, user_id=0dfb161b6a9144f9aa0d4cf47d90ab20, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:12.560819 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d92910-7705-4925-9b43-585793a91cd1 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:12.579057 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:12.590238 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 795/1599] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:12 2022] PUT /identity/v3/projects/4370999c56184197856c3b8b25fd2e20/users/0dfb161b6a9144f9aa0d4cf47d90ab20/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:12.594317 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d92910-7705-4925-9b43-585793a91cd1 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:12.636611 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec60623-7747-4990-9b3f-8f1dcb000dbb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:12.638892 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec60623-7747-4990-9b3f-8f1dcb000dbb service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:12.652993 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 805/1600] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:12 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:12.654352 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfad3d2-4512-4a4e-b91f-af9fadb85811 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:12.654696 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfad3d2-4512-4a4e-b91f-af9fadb85811 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:12.655076 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfad3d2-4512-4a4e-b91f-af9fadb85811 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:12.719543 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-5dfad3d2-4512-4a4e-b91f-af9fadb85811 None None] MFA Rules not processed for user `0dfb161b6a9144f9aa0d4cf47d90ab20`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:12.721102 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-5dfad3d2-4512-4a4e-b91f-af9fadb85811 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:12.738585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d92910-7705-4925-9b43-585793a91cd1 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:12.764942 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2646b5-11cd-487a-b928-574acb3b6486 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:12.765695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2646b5-11cd-487a-b928-574acb3b6486 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:12.766169 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2646b5-11cd-487a-b928-574acb3b6486 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:12.766507 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2646b5-11cd-487a-b928-574acb3b6486 admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:12.771886 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2646b5-11cd-487a-b928-574acb3b6486 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:58:12.772283 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2646b5-11cd-487a-b928-574acb3b6486 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:12.774470 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2646b5-11cd-487a-b928-574acb3b6486 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:12.803139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 796/1601] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:58:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:58:12.806605 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2646b5-11cd-487a-b928-574acb3b6486 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:12.811300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-bd2646b5-11cd-487a-b928-574acb3b6486 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:12.829332 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 806/1602] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:12 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:12.830436 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfad3d2-4512-4a4e-b91f-af9fadb85811 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:12.845623 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfad3d2-4512-4a4e-b91f-af9fadb85811 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:12.850848 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:DELETE=6,keystone:SELECT=92,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:12.851014 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-374b04a2-6a97-42c9-96a6-ab5a1f689a31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:12.851884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374b04a2-6a97-42c9-96a6-ab5a1f689a31 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:12.852238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374b04a2-6a97-42c9-96a6-ab5a1f689a31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:12.852594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374b04a2-6a97-42c9-96a6-ab5a1f689a31 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:12.853384 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-374b04a2-6a97-42c9-96a6-ab5a1f689a31 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:12.855772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-374b04a2-6a97-42c9-96a6-ab5a1f689a31 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:12.907387 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 807/1603] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:58:12 2022] POST /identity/v3/projects => generated 365 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:12.912927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-374b04a2-6a97-42c9-96a6-ab5a1f689a31 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:12.935632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2646b5-11cd-487a-b928-574acb3b6486 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:13.008968 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-374b04a2-6a97-42c9-96a6-ab5a1f689a31 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:13.014421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a43db56-f66f-45e4-b54e-25c1b30de4b0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:13.015416 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a43db56-f66f-45e4-b54e-25c1b30de4b0 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:13.016774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a43db56-f66f-45e4-b54e-25c1b30de4b0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:13.017140 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a43db56-f66f-45e4-b54e-25c1b30de4b0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:13.029775 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b44ad79-66e8-4705-a564-604fb31901c6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:13.030437 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b44ad79-66e8-4705-a564-604fb31901c6 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:13.030895 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b44ad79-66e8-4705-a564-604fb31901c6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:13.031259 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b44ad79-66e8-4705-a564-604fb31901c6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:13.033440 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-1b44ad79-66e8-4705-a564-604fb31901c6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:13.076677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:13.091478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a43db56-f66f-45e4-b54e-25c1b30de4b0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:13.093336 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a43db56-f66f-45e4-b54e-25c1b30de4b0 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:13.106065 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 808/1604] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:12 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:13.118757 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a43db56-f66f-45e4-b54e-25c1b30de4b0 service cinder] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:13.138850 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b44ad79-66e8-4705-a564-604fb31901c6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:13.140415 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b44ad79-66e8-4705-a564-604fb31901c6 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:13.176333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 797/1605] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:12 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:13.254913 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a43db56-f66f-45e4-b54e-25c1b30de4b0 service cinder] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:13.262705 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6916c9-006a-48f5-b948-5e4eadbb67f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:13.264106 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6916c9-006a-48f5-b948-5e4eadbb67f6 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:13.264220 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6916c9-006a-48f5-b948-5e4eadbb67f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:13.264443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6916c9-006a-48f5-b948-5e4eadbb67f6 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:13.266045 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6916c9-006a-48f5-b948-5e4eadbb67f6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:13.268020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6916c9-006a-48f5-b948-5e4eadbb67f6 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:13.309634 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 809/1606] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:58:13 2022] POST /identity/v3/users => generated 283 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:13.314483 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b44ad79-66e8-4705-a564-604fb31901c6 service cinder] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:13.332054 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b44ad79-66e8-4705-a564-604fb31901c6 service cinder] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:13.348978 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1222cf91-2055-447b-8402-5a999551a847 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:13.349687 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1222cf91-2055-447b-8402-5a999551a847 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:13.350015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1222cf91-2055-447b-8402-5a999551a847 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:13.350348 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1222cf91-2055-447b-8402-5a999551a847 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:13.351078 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1222cf91-2055-447b-8402-5a999551a847 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:13.353449 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1222cf91-2055-447b-8402-5a999551a847 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:13.378442 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6916c9-006a-48f5-b948-5e4eadbb67f6 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:13.378581 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 798/1607] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:13 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:13.391844 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6916c9-006a-48f5-b948-5e4eadbb67f6 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:13.404573 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e61107-4631-4e7a-9a8a-da102e6610b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:13.406834 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e61107-4631-4e7a-9a8a-da102e6610b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:13.407230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e61107-4631-4e7a-9a8a-da102e6610b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:13.408489 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e61107-4631-4e7a-9a8a-da102e6610b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:13.410634 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e61107-4631-4e7a-9a8a-da102e6610b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:13.414251 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e61107-4631-4e7a-9a8a-da102e6610b4 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:13.445270 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 810/1608] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:13 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 00:58:13.446465 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1222cf91-2055-447b-8402-5a999551a847 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:13.468578 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1222cf91-2055-447b-8402-5a999551a847 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:13.482644 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177f1e64-8dd5-41dc-893e-46b053acea3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:13.483332 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177f1e64-8dd5-41dc-893e-46b053acea3d admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:13.483629 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177f1e64-8dd5-41dc-893e-46b053acea3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:13.483961 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177f1e64-8dd5-41dc-893e-46b053acea3d admin admin] PATH_INFO: `/v3/projects/776d07aafb084e2e9517aedc639f012a/users/e503fffc519d4948985c6f2d75265e19/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:13.521340 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177f1e64-8dd5-41dc-893e-46b053acea3d admin admin] RBAC: Authorizing `identity:create_grant(project_id=776d07aafb084e2e9517aedc639f012a, user_id=e503fffc519d4948985c6f2d75265e19, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:13.524320 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177f1e64-8dd5-41dc-893e-46b053acea3d admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:13.559960 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 799/1609] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:13 2022] PUT /identity/v3/projects/776d07aafb084e2e9517aedc639f012a/users/e503fffc519d4948985c6f2d75265e19/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:13.568045 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faec34ff-99b8-4e65-ae9d-8afedf1a2675 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:13.568567 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faec34ff-99b8-4e65-ae9d-8afedf1a2675 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:13.569056 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faec34ff-99b8-4e65-ae9d-8afedf1a2675 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:13.620785 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177f1e64-8dd5-41dc-893e-46b053acea3d admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:13.623045 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-177f1e64-8dd5-41dc-893e-46b053acea3d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:13.637449 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-faec34ff-99b8-4e65-ae9d-8afedf1a2675 None None] MFA Rules not processed for user `e503fffc519d4948985c6f2d75265e19`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:13.638896 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-faec34ff-99b8-4e65-ae9d-8afedf1a2675 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:13.759063 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:13.805818 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177f1e64-8dd5-41dc-893e-46b053acea3d admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:13.811680 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598b6199-9b9c-4a9b-b5ae-1909d2a23523 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:13.812735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598b6199-9b9c-4a9b-b5ae-1909d2a23523 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:13.813146 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598b6199-9b9c-4a9b-b5ae-1909d2a23523 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:13.813563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598b6199-9b9c-4a9b-b5ae-1909d2a23523 admin admin] PATH_INFO: `/v3/users/b3473287e6224664bc645a8f442c6176` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:13.814912 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598b6199-9b9c-4a9b-b5ae-1909d2a23523 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b3473287e6224664bc645a8f442c6176)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:13.817040 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598b6199-9b9c-4a9b-b5ae-1909d2a23523 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:13.824104 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 811/1610] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:13 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:13.828541 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faec34ff-99b8-4e65-ae9d-8afedf1a2675 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:13.844613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faec34ff-99b8-4e65-ae9d-8afedf1a2675 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:13.851802 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d90020-614f-49c5-b8a2-5f93fd33734d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:13.852357 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d90020-614f-49c5-b8a2-5f93fd33734d admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:13.852606 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d90020-614f-49c5-b8a2-5f93fd33734d admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:13.853009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d90020-614f-49c5-b8a2-5f93fd33734d admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:13.853933 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d90020-614f-49c5-b8a2-5f93fd33734d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:13.855669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d90020-614f-49c5-b8a2-5f93fd33734d admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:13.884901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-598b6199-9b9c-4a9b-b5ae-1909d2a23523 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b3473287e6224664bc645a8f442c6176'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:13.889175 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 812/1611] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:58:13 2022] POST /identity/v3/projects => generated 365 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:58:13.893598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02d8c13-2c7e-4ef6-8081-833223a66890 None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:13.894456 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02d8c13-2c7e-4ef6-8081-833223a66890 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:13.894877 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02d8c13-2c7e-4ef6-8081-833223a66890 None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:13.895591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-598b6199-9b9c-4a9b-b5ae-1909d2a23523 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b3473287e6224664bc645a8f442c6176'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:13.896268 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-598b6199-9b9c-4a9b-b5ae-1909d2a23523 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b3473287e6224664bc645a8f442c6176'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:13.896919 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 813/1612] 10.222.0.16 () {54 vars in 921 bytes} [Sun Oct 30 00:58:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:58:13.897528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02d8c13-2c7e-4ef6-8081-833223a66890 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:13.899310 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e02d8c13-2c7e-4ef6-8081-833223a66890 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:13.905321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-598b6199-9b9c-4a9b-b5ae-1909d2a23523 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b3473287e6224664bc645a8f442c6176'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:13.922112 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-598b6199-9b9c-4a9b-b5ae-1909d2a23523 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b3473287e6224664bc645a8f442c6176'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:13.929569 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 800/1613] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:58:13 2022] DELETE /identity/v3/users/b3473287e6224664bc645a8f442c6176 => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:13.933970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12261cce-ee8a-481f-9676-2b8d3d7317c1 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:13.934247 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12261cce-ee8a-481f-9676-2b8d3d7317c1 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:13.934516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12261cce-ee8a-481f-9676-2b8d3d7317c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:13.948110 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:13.956892 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-12261cce-ee8a-481f-9676-2b8d3d7317c1 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:13.957801 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-12261cce-ee8a-481f-9676-2b8d3d7317c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:13.983436 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02d8c13-2c7e-4ef6-8081-833223a66890 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:14.022004 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 801/1614] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:14.033168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12261cce-ee8a-481f-9676-2b8d3d7317c1 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:14.035470 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-12261cce-ee8a-481f-9676-2b8d3d7317c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:14.054105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea741312-4500-4d7c-8643-7518e69f6f61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:14.054757 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea741312-4500-4d7c-8643-7518e69f6f61 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:14.055120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea741312-4500-4d7c-8643-7518e69f6f61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:14.055525 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea741312-4500-4d7c-8643-7518e69f6f61 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:14.056291 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea741312-4500-4d7c-8643-7518e69f6f61 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:14.058637 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea741312-4500-4d7c-8643-7518e69f6f61 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:14.088853 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 814/1615] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:58:13 2022] POST /identity/v3/users => generated 289 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:14.095404 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea741312-4500-4d7c-8643-7518e69f6f61 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:14.110650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea741312-4500-4d7c-8643-7518e69f6f61 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:14.114652 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12261cce-ee8a-481f-9676-2b8d3d7317c1 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:14.116273 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706b4d50-7285-4340-aae5-78f483ddac8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:14.117253 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706b4d50-7285-4340-aae5-78f483ddac8e admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:14.117605 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706b4d50-7285-4340-aae5-78f483ddac8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:14.118603 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706b4d50-7285-4340-aae5-78f483ddac8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:14.119606 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706b4d50-7285-4340-aae5-78f483ddac8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:14.120048 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e15f86-7feb-41c7-a419-911da6821054 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:14.120936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e15f86-7feb-41c7-a419-911da6821054 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:14.121384 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e15f86-7feb-41c7-a419-911da6821054 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:14.121705 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706b4d50-7285-4340-aae5-78f483ddac8e admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:14.121997 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e15f86-7feb-41c7-a419-911da6821054 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:14.134539 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e15f86-7feb-41c7-a419-911da6821054 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:14.136782 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e15f86-7feb-41c7-a419-911da6821054 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:14.142270 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 815/1616] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:14 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:14.146582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706b4d50-7285-4340-aae5-78f483ddac8e admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:14.155971 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 802/1617] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 00:58:14.157223 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9544219e-bfd2-4d97-ab01-c743717300bd None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:14.157498 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9544219e-bfd2-4d97-ab01-c743717300bd None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:14.157822 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9544219e-bfd2-4d97-ab01-c743717300bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:14.166710 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706b4d50-7285-4340-aae5-78f483ddac8e admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:14.176900 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953d846c-4c56-49bb-b75b-99fac903818c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:14.176900 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953d846c-4c56-49bb-b75b-99fac903818c admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:14.176900 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953d846c-4c56-49bb-b75b-99fac903818c admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:14.176900 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953d846c-4c56-49bb-b75b-99fac903818c admin admin] PATH_INFO: `/v3/projects/3449cc37b0c04f3c8414acc1b9a386e7/users/ec15a942520144e9917c275c5272e0c1/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:14.180961 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-9544219e-bfd2-4d97-ab01-c743717300bd None None] MFA Rules not processed for user `63cc035057064805a95e9f12714ebade`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:14.184199 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-9544219e-bfd2-4d97-ab01-c743717300bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:14.196135 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953d846c-4c56-49bb-b75b-99fac903818c admin admin] RBAC: Authorizing `identity:create_grant(project_id=3449cc37b0c04f3c8414acc1b9a386e7, user_id=ec15a942520144e9917c275c5272e0c1, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:14.200316 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953d846c-4c56-49bb-b75b-99fac903818c admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:14.200617 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:14.242817 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 816/1618] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:14 2022] PUT /identity/v3/projects/3449cc37b0c04f3c8414acc1b9a386e7/users/ec15a942520144e9917c275c5272e0c1/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 00:58:14.242817 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953d846c-4c56-49bb-b75b-99fac903818c admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:14.246794 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-953d846c-4c56-49bb-b75b-99fac903818c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:14.309084 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 803/1619] 10.222.0.16 () {58 vars in 1018 bytes} [Sun Oct 30 00:58:14 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:14.310849 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9544219e-bfd2-4d97-ab01-c743717300bd None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:14.333054 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9544219e-bfd2-4d97-ab01-c743717300bd None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:14.358160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953d846c-4c56-49bb-b75b-99fac903818c admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:14.371279 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc80fd1a-d40d-48b8-8ed8-76911a8a02aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:14.371995 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc80fd1a-d40d-48b8-8ed8-76911a8a02aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:14.372270 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc80fd1a-d40d-48b8-8ed8-76911a8a02aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:14.372539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc80fd1a-d40d-48b8-8ed8-76911a8a02aa admin admin] PATH_INFO: `/v3/users/c92f7b4d09d74b8d93d739c1938694e6` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:14.373778 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc80fd1a-d40d-48b8-8ed8-76911a8a02aa admin admin] RBAC: Authorizing `identity:delete_user(user_id=c92f7b4d09d74b8d93d739c1938694e6)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:14.375931 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc80fd1a-d40d-48b8-8ed8-76911a8a02aa admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:14.421448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90bf167-cfb3-4bac-a3c9-8807e09cf942 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:14.422390 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90bf167-cfb3-4bac-a3c9-8807e09cf942 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:14.422390 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90bf167-cfb3-4bac-a3c9-8807e09cf942 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:14.422615 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90bf167-cfb3-4bac-a3c9-8807e09cf942 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:14.423223 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90bf167-cfb3-4bac-a3c9-8807e09cf942 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:14.424839 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90bf167-cfb3-4bac-a3c9-8807e09cf942 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:14.430971 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=91,keystone:INSERT=4,keystone:DELETE=5 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:14.437214 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 804/1620] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:14 2022] GET /identity/v3/roles => generated 1420 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:14.438998 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d6eafe-a915-421a-b3dc-ff89c53e2532 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:14.439457 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d6eafe-a915-421a-b3dc-ff89c53e2532 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:14.439982 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d6eafe-a915-421a-b3dc-ff89c53e2532 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:14.441861 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 805/1621] 10.222.0.16 () {54 vars in 961 bytes} [Sun Oct 30 00:58:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 00:58:14.442208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-cc80fd1a-d40d-48b8-8ed8-76911a8a02aa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c92f7b4d09d74b8d93d739c1938694e6'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:14.442977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d6eafe-a915-421a-b3dc-ff89c53e2532 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:14.455636 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-cc80fd1a-d40d-48b8-8ed8-76911a8a02aa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c92f7b4d09d74b8d93d739c1938694e6'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:14.456552 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-cc80fd1a-d40d-48b8-8ed8-76911a8a02aa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c92f7b4d09d74b8d93d739c1938694e6'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:14.458715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d6eafe-a915-421a-b3dc-ff89c53e2532 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:14.464081 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b665d6de-23bf-47fd-b21f-558688d029ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:14.464975 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b665d6de-23bf-47fd-b21f-558688d029ba admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:14.465243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b665d6de-23bf-47fd-b21f-558688d029ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:14.465512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b665d6de-23bf-47fd-b21f-558688d029ba admin admin] PATH_INFO: `/v3/projects/3449cc37b0c04f3c8414acc1b9a386e7/users/ec15a942520144e9917c275c5272e0c1/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:14.468665 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b665d6de-23bf-47fd-b21f-558688d029ba admin admin] RBAC: Authorizing `identity:create_grant(project_id=3449cc37b0c04f3c8414acc1b9a386e7, user_id=ec15a942520144e9917c275c5272e0c1, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:14.470309 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-cc80fd1a-d40d-48b8-8ed8-76911a8a02aa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c92f7b4d09d74b8d93d739c1938694e6'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:14.471542 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b665d6de-23bf-47fd-b21f-558688d029ba admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:14.477791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-cc80fd1a-d40d-48b8-8ed8-76911a8a02aa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c92f7b4d09d74b8d93d739c1938694e6'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:14.482273 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 806/1622] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:14 2022] PUT /identity/v3/projects/3449cc37b0c04f3c8414acc1b9a386e7/users/ec15a942520144e9917c275c5272e0c1/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:14.485135 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 817/1623] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:58:14 2022] DELETE /identity/v3/users/c92f7b4d09d74b8d93d739c1938694e6 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:14.493027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470739a7-0fe0-4854-9ad8-5935284dfb87 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:14.493313 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470739a7-0fe0-4854-9ad8-5935284dfb87 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:14.493586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470739a7-0fe0-4854-9ad8-5935284dfb87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:14.499990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899c0e8a-429b-4787-b955-e4bf9f1dc018 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:14.501328 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899c0e8a-429b-4787-b955-e4bf9f1dc018 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:14.502589 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899c0e8a-429b-4787-b955-e4bf9f1dc018 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:14.519706 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-470739a7-0fe0-4854-9ad8-5935284dfb87 None None] MFA Rules not processed for user `63cc035057064805a95e9f12714ebade`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:14.521561 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-470739a7-0fe0-4854-9ad8-5935284dfb87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:14.553524 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-899c0e8a-429b-4787-b955-e4bf9f1dc018 None None] MFA Rules not processed for user `ec15a942520144e9917c275c5272e0c1`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:14.556111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-899c0e8a-429b-4787-b955-e4bf9f1dc018 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:14.603711 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 807/1624] 10.222.0.16 () {58 vars in 1058 bytes} [Sun Oct 30 00:58:14 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:14.604619 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470739a7-0fe0-4854-9ad8-5935284dfb87 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:14.606085 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-470739a7-0fe0-4854-9ad8-5935284dfb87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:14.642306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:14.652255 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 818/1625] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:14 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:14.660236 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899c0e8a-429b-4787-b955-e4bf9f1dc018 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:14.673459 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899c0e8a-429b-4787-b955-e4bf9f1dc018 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:14.680244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c8ecad-8e18-4940-ad1e-9e2c44449d52 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '63cc035057064805a95e9f12714ebade', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:14.680869 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c8ecad-8e18-4940-ad1e-9e2c44449d52 service placement] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:14.681150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c8ecad-8e18-4940-ad1e-9e2c44449d52 service placement] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:14.681435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c8ecad-8e18-4940-ad1e-9e2c44449d52 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:14.682329 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470739a7-0fe0-4854-9ad8-5935284dfb87 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:14.683544 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-19c8ecad-8e18-4940-ad1e-9e2c44449d52 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:14.687862 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813f31c8-e482-4538-bac8-33c760c845dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:14.688614 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813f31c8-e482-4538-bac8-33c760c845dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:14.688785 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813f31c8-e482-4538-bac8-33c760c845dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:14.689056 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813f31c8-e482-4538-bac8-33c760c845dc admin admin] PATH_INFO: `/v3/projects/e59bb9a40a29499f916536e02ba1c238` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:14.690193 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813f31c8-e482-4538-bac8-33c760c845dc admin admin] RBAC: Authorizing `identity:delete_project(project_id=e59bb9a40a29499f916536e02ba1c238)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:14.692161 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813f31c8-e482-4538-bac8-33c760c845dc admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:14.698230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c8ecad-8e18-4940-ad1e-9e2c44449d52 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:14.699863 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c8ecad-8e18-4940-ad1e-9e2c44449d52 service placement] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:14.711187 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 819/1626] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:14 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:14.712424 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c8ecad-8e18-4940-ad1e-9e2c44449d52 service placement] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:14.714054 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-19c8ecad-8e18-4940-ad1e-9e2c44449d52 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:14.759525 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-813f31c8-e482-4538-bac8-33c760c845dc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e59bb9a40a29499f916536e02ba1c238', 'request_id': 'req-813f31c8-e482-4538-bac8-33c760c845dc'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:14.766498 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-813f31c8-e482-4538-bac8-33c760c845dc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e59bb9a40a29499f916536e02ba1c238', 'request_id': 'req-813f31c8-e482-4538-bac8-33c760c845dc'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:14.776272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-813f31c8-e482-4538-bac8-33c760c845dc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e59bb9a40a29499f916536e02ba1c238', 'request_id': 'req-813f31c8-e482-4538-bac8-33c760c845dc'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:14.785462 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c8ecad-8e18-4940-ad1e-9e2c44449d52 service placement] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:14.797399 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-813f31c8-e482-4538-bac8-33c760c845dc admin admin] The token cache is being invalidate because project e59bb9a40a29499f916536e02ba1c238 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:58:14.797399 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-813f31c8-e482-4538-bac8-33c760c845dc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:14.797399 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-813f31c8-e482-4538-bac8-33c760c845dc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:14.797399 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 808/1627] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:58:14 2022] DELETE /identity/v3/projects/e59bb9a40a29499f916536e02ba1c238 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:14.798492 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813f31c8-e482-4538-bac8-33c760c845dc admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:14.800110 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-813f31c8-e482-4538-bac8-33c760c845dc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:14.833234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:DELETE=5,keystone:SELECT=94,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:14.843270 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb547174-8000-4ea9-970a-71d07784f833 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:14.844129 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb547174-8000-4ea9-970a-71d07784f833 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:14.844746 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb547174-8000-4ea9-970a-71d07784f833 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:14.845227 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb547174-8000-4ea9-970a-71d07784f833 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:14.847297 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-bb547174-8000-4ea9-970a-71d07784f833 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:14.874886 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813f31c8-e482-4538-bac8-33c760c845dc admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:14.879940 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef38fd2d-8e54-4234-80bf-010656c250c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:14.881174 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef38fd2d-8e54-4234-80bf-010656c250c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:14.881734 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef38fd2d-8e54-4234-80bf-010656c250c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:14.882011 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef38fd2d-8e54-4234-80bf-010656c250c5 admin admin] PATH_INFO: `/v3/projects/0f7d505c7b9e4ca08d6437e6e529a039` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:14.883159 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef38fd2d-8e54-4234-80bf-010656c250c5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f7d505c7b9e4ca08d6437e6e529a039)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:14.885217 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef38fd2d-8e54-4234-80bf-010656c250c5 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:14.917597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb547174-8000-4ea9-970a-71d07784f833 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:14.919282 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb547174-8000-4ea9-970a-71d07784f833 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:14.931368 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 820/1628] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:14 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:14.932710 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efed6cec-d419-48db-8e92-c651a1d5164f None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:14.932944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efed6cec-d419-48db-8e92-c651a1d5164f None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:14.933215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efed6cec-d419-48db-8e92-c651a1d5164f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:14.938776 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-ef38fd2d-8e54-4234-80bf-010656c250c5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f7d505c7b9e4ca08d6437e6e529a039', 'request_id': 'req-ef38fd2d-8e54-4234-80bf-010656c250c5'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:14.946091 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-ef38fd2d-8e54-4234-80bf-010656c250c5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f7d505c7b9e4ca08d6437e6e529a039', 'request_id': 'req-ef38fd2d-8e54-4234-80bf-010656c250c5'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:14.948545 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:14.959281 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-ef38fd2d-8e54-4234-80bf-010656c250c5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f7d505c7b9e4ca08d6437e6e529a039', 'request_id': 'req-ef38fd2d-8e54-4234-80bf-010656c250c5'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:14.964259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-efed6cec-d419-48db-8e92-c651a1d5164f None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:14.965829 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-efed6cec-d419-48db-8e92-c651a1d5164f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:14.974530 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-ef38fd2d-8e54-4234-80bf-010656c250c5 admin admin] The token cache is being invalidate because project 0f7d505c7b9e4ca08d6437e6e529a039 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:58:14.974884 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-ef38fd2d-8e54-4234-80bf-010656c250c5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:14.975763 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-ef38fd2d-8e54-4234-80bf-010656c250c5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:14.977479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 809/1629] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:58:14 2022] DELETE /identity/v3/projects/0f7d505c7b9e4ca08d6437e6e529a039 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:14.989004 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35051d6-dcc0-4e4c-a71f-84cc93449ed5 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:14.989287 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35051d6-dcc0-4e4c-a71f-84cc93449ed5 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:14.989557 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35051d6-dcc0-4e4c-a71f-84cc93449ed5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:15.012816 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-b35051d6-dcc0-4e4c-a71f-84cc93449ed5 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:15.013998 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b35051d6-dcc0-4e4c-a71f-84cc93449ed5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:15.061570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 821/1630] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:15.065241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b38cb82-43d8-4407-aab0-23868d25cf72 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:15.065532 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b38cb82-43d8-4407-aab0-23868d25cf72 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:15.065805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b38cb82-43d8-4407-aab0-23868d25cf72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:15.098459 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-3b38cb82-43d8-4407-aab0-23868d25cf72 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:15.099949 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3b38cb82-43d8-4407-aab0-23868d25cf72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:15.117339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 822/1631] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:15.118239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b38cb82-43d8-4407-aab0-23868d25cf72 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:15.122612 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 810/1632] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:15.129643 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b38cb82-43d8-4407-aab0-23868d25cf72 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:15.130964 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6725bb62-d2ff-49d5-8b38-3a58094c9c94 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:15.131248 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6725bb62-d2ff-49d5-8b38-3a58094c9c94 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:15.131546 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6725bb62-d2ff-49d5-8b38-3a58094c9c94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:15.139324 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f998f8-9dd3-4f8a-ac1a-125464cb58ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:15.140587 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f998f8-9dd3-4f8a-ac1a-125464cb58ed admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:15.141001 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f998f8-9dd3-4f8a-ac1a-125464cb58ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:15.141562 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f998f8-9dd3-4f8a-ac1a-125464cb58ed admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:15.143324 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f998f8-9dd3-4f8a-ac1a-125464cb58ed admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:58:15.143324 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f998f8-9dd3-4f8a-ac1a-125464cb58ed admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:15.145103 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f998f8-9dd3-4f8a-ac1a-125464cb58ed admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:15.154920 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-6725bb62-d2ff-49d5-8b38-3a58094c9c94 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:15.156014 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-6725bb62-d2ff-49d5-8b38-3a58094c9c94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:15.161161 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 823/1633] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:58:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:58:15.165778 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f998f8-9dd3-4f8a-ac1a-125464cb58ed admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:15.166698 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 811/1634] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:58:15.170768 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6725bb62-d2ff-49d5-8b38-3a58094c9c94 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:15.180199 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f998f8-9dd3-4f8a-ac1a-125464cb58ed admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:15.181958 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6725bb62-d2ff-49d5-8b38-3a58094c9c94 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:15.187819 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078a5ac4-447b-42cd-bad4-fa0e055455f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:15.188180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90602a33-6e6b-4ab4-b7c9-f0b2d6b3e050 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:15.188487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078a5ac4-447b-42cd-bad4-fa0e055455f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:15.188787 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90602a33-6e6b-4ab4-b7c9-f0b2d6b3e050 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:15.188888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078a5ac4-447b-42cd-bad4-fa0e055455f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:15.189082 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90602a33-6e6b-4ab4-b7c9-f0b2d6b3e050 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:15.189177 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078a5ac4-447b-42cd-bad4-fa0e055455f9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:15.189350 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90602a33-6e6b-4ab4-b7c9-f0b2d6b3e050 admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:15.189724 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078a5ac4-447b-42cd-bad4-fa0e055455f9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:15.189994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90602a33-6e6b-4ab4-b7c9-f0b2d6b3e050 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:58:15.190292 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90602a33-6e6b-4ab4-b7c9-f0b2d6b3e050 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:15.191614 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078a5ac4-447b-42cd-bad4-fa0e055455f9 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:15.191892 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90602a33-6e6b-4ab4-b7c9-f0b2d6b3e050 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:15.207751 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 812/1635] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:58:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:58:15.209846 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90602a33-6e6b-4ab4-b7c9-f0b2d6b3e050 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:15.222120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 824/1636] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:58:15 2022] POST /identity/v3/projects => generated 381 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:58:15.223882 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90602a33-6e6b-4ab4-b7c9-f0b2d6b3e050 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:15.226217 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078a5ac4-447b-42cd-bad4-fa0e055455f9 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:15.296133 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a71ada8-52ee-4ebc-8075-0f0639a17761 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:15.296819 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a71ada8-52ee-4ebc-8075-0f0639a17761 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:15.297018 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a71ada8-52ee-4ebc-8075-0f0639a17761 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:15.297286 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a71ada8-52ee-4ebc-8075-0f0639a17761 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:15.298002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a71ada8-52ee-4ebc-8075-0f0639a17761 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:15.300232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a71ada8-52ee-4ebc-8075-0f0639a17761 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:15.303545 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:15.318459 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078a5ac4-447b-42cd-bad4-fa0e055455f9 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:15.325434 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6643d26-a9a5-4933-aaf2-9e67696e3495 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:15.326076 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6643d26-a9a5-4933-aaf2-9e67696e3495 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:15.326392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6643d26-a9a5-4933-aaf2-9e67696e3495 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:15.326630 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6643d26-a9a5-4933-aaf2-9e67696e3495 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:15.327279 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6643d26-a9a5-4933-aaf2-9e67696e3495 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:15.329318 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6643d26-a9a5-4933-aaf2-9e67696e3495 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:15.337191 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 813/1637] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:58:15 2022] POST /identity/v3/projects => generated 391 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:15.338407 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a71ada8-52ee-4ebc-8075-0f0639a17761 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:15.357138 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 825/1638] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:58:15 2022] POST /identity/v3/users => generated 291 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:15.361523 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6643d26-a9a5-4933-aaf2-9e67696e3495 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:15.378312 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6643d26-a9a5-4933-aaf2-9e67696e3495 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:15.405086 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:INSERT=2,keystone:DELETE=1,keystone:SELECT=97 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:15.417169 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a71ada8-52ee-4ebc-8075-0f0639a17761 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:15.424104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa235650-02f0-4a4e-9a8d-9a004b213de2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:15.424783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa235650-02f0-4a4e-9a8d-9a004b213de2 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:15.425119 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa235650-02f0-4a4e-9a8d-9a004b213de2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:15.425447 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa235650-02f0-4a4e-9a8d-9a004b213de2 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:15.426557 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa235650-02f0-4a4e-9a8d-9a004b213de2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:15.428654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa235650-02f0-4a4e-9a8d-9a004b213de2 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:15.456679 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 814/1639] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:58:15 2022] POST /identity/v3/users => generated 296 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:15.463121 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8daae52f-a9d0-485b-8f3f-653f92aa635e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:15.463790 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daae52f-a9d0-485b-8f3f-653f92aa635e admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:15.464257 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daae52f-a9d0-485b-8f3f-653f92aa635e admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:15.464527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daae52f-a9d0-485b-8f3f-653f92aa635e admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:15.465182 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8daae52f-a9d0-485b-8f3f-653f92aa635e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:15.466884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8daae52f-a9d0-485b-8f3f-653f92aa635e admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:15.470756 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa235650-02f0-4a4e-9a8d-9a004b213de2 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:15.479843 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 826/1640] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:15 2022] GET /identity/v3/roles => generated 1420 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 00:58:15.484359 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8daae52f-a9d0-485b-8f3f-653f92aa635e admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:15.486764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa235650-02f0-4a4e-9a8d-9a004b213de2 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:15.492759 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e17b14-be3b-476a-af5c-63d34dc19d5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:15.493672 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e17b14-be3b-476a-af5c-63d34dc19d5c admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:15.494146 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e17b14-be3b-476a-af5c-63d34dc19d5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:15.494387 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e17b14-be3b-476a-af5c-63d34dc19d5c admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:15.495196 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e17b14-be3b-476a-af5c-63d34dc19d5c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:15.497650 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e17b14-be3b-476a-af5c-63d34dc19d5c admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:15.499629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8daae52f-a9d0-485b-8f3f-653f92aa635e admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:15.504308 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21050b56-5623-4fe3-8c94-65b320a6ef56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:15.504927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21050b56-5623-4fe3-8c94-65b320a6ef56 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:15.505200 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21050b56-5623-4fe3-8c94-65b320a6ef56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:15.505475 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21050b56-5623-4fe3-8c94-65b320a6ef56 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:15.506110 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21050b56-5623-4fe3-8c94-65b320a6ef56 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:15.507910 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21050b56-5623-4fe3-8c94-65b320a6ef56 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:15.512196 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 815/1641] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:15 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:15.512196 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e17b14-be3b-476a-af5c-63d34dc19d5c admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:15.512196 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-45e17b14-be3b-476a-af5c-63d34dc19d5c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:15.523118 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 827/1642] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:15 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:15.523118 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21050b56-5623-4fe3-8c94-65b320a6ef56 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:15.527376 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e17b14-be3b-476a-af5c-63d34dc19d5c admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:15.530962 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21050b56-5623-4fe3-8c94-65b320a6ef56 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:15.532046 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847b477d-f9f6-451d-8b59-1be1ee087097 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:15.532680 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847b477d-f9f6-451d-8b59-1be1ee087097 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:15.532957 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847b477d-f9f6-451d-8b59-1be1ee087097 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:15.533224 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847b477d-f9f6-451d-8b59-1be1ee087097 admin admin] PATH_INFO: `/v3/users/e503fffc519d4948985c6f2d75265e19` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:15.534360 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847b477d-f9f6-451d-8b59-1be1ee087097 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e503fffc519d4948985c6f2d75265e19)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:15.535456 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8865083d-a48f-4a41-80e5-7dc9960b7d4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:15.536084 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8865083d-a48f-4a41-80e5-7dc9960b7d4f admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:15.536314 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847b477d-f9f6-451d-8b59-1be1ee087097 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:15.536409 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8865083d-a48f-4a41-80e5-7dc9960b7d4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:15.536630 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8865083d-a48f-4a41-80e5-7dc9960b7d4f admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:15.537247 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8865083d-a48f-4a41-80e5-7dc9960b7d4f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:15.539041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8865083d-a48f-4a41-80e5-7dc9960b7d4f admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:15.549387 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 828/1643] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:15 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:15.550180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8865083d-a48f-4a41-80e5-7dc9960b7d4f admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:15.560022 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8865083d-a48f-4a41-80e5-7dc9960b7d4f admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:15.564932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e91b808-309e-48b7-a1aa-8246566e0159 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:15.565576 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e91b808-309e-48b7-a1aa-8246566e0159 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:15.565849 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e91b808-309e-48b7-a1aa-8246566e0159 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:15.566119 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e91b808-309e-48b7-a1aa-8246566e0159 admin admin] PATH_INFO: `/v3/projects/2fbc73a853584aaba6109090bd6b1281/users/ddeaa7a90ee7483590e6bc0d189e1e2a/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:15.581343 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e91b808-309e-48b7-a1aa-8246566e0159 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2fbc73a853584aaba6109090bd6b1281, user_id=ddeaa7a90ee7483590e6bc0d189e1e2a, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:15.583931 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e91b808-309e-48b7-a1aa-8246566e0159 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:15.587848 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-847b477d-f9f6-451d-8b59-1be1ee087097 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e503fffc519d4948985c6f2d75265e19'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:15.597423 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-847b477d-f9f6-451d-8b59-1be1ee087097 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e503fffc519d4948985c6f2d75265e19'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:15.598056 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-847b477d-f9f6-451d-8b59-1be1ee087097 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e503fffc519d4948985c6f2d75265e19'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:15.599150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 829/1644] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:15 2022] PUT /identity/v3/projects/2fbc73a853584aaba6109090bd6b1281/users/ddeaa7a90ee7483590e6bc0d189e1e2a/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:15.599912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e91b808-309e-48b7-a1aa-8246566e0159 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:15.601615 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-4e91b808-309e-48b7-a1aa-8246566e0159 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:15.609090 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-847b477d-f9f6-451d-8b59-1be1ee087097 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e503fffc519d4948985c6f2d75265e19'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:15.620867 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-847b477d-f9f6-451d-8b59-1be1ee087097 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e503fffc519d4948985c6f2d75265e19'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:15.631695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 816/1645] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:58:15 2022] DELETE /identity/v3/users/e503fffc519d4948985c6f2d75265e19 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:15.631695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7234e08d-14b2-4387-978c-cd6403cae3d1 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:15.631695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7234e08d-14b2-4387-978c-cd6403cae3d1 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:15.631695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7234e08d-14b2-4387-978c-cd6403cae3d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:15.653250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-7234e08d-14b2-4387-978c-cd6403cae3d1 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:15.655129 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-7234e08d-14b2-4387-978c-cd6403cae3d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:15.673005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e91b808-309e-48b7-a1aa-8246566e0159 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:15.714380 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:15.721394 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 817/1646] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:15.722704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1365e65c-4455-4636-a768-22882ea4eeb7 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:15.722987 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1365e65c-4455-4636-a768-22882ea4eeb7 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:15.723258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1365e65c-4455-4636-a768-22882ea4eeb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:15.739630 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e670f16-913b-4af5-a4f3-c631f368817b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:15.740606 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e670f16-913b-4af5-a4f3-c631f368817b admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:15.740879 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e670f16-913b-4af5-a4f3-c631f368817b admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:15.741141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e670f16-913b-4af5-a4f3-c631f368817b admin admin] PATH_INFO: `/v3/projects/f48276ac51dd4bb1912fc670433cf6c4/users/31b7342ecf07408cab6bb040f361fa1f/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:15.743802 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-1365e65c-4455-4636-a768-22882ea4eeb7 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:15.745018 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-1365e65c-4455-4636-a768-22882ea4eeb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:15.752657 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 818/1647] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:15.753452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1365e65c-4455-4636-a768-22882ea4eeb7 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:15.754871 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-1365e65c-4455-4636-a768-22882ea4eeb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:15.756221 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e670f16-913b-4af5-a4f3-c631f368817b admin admin] RBAC: Authorizing `identity:create_grant(project_id=f48276ac51dd4bb1912fc670433cf6c4, user_id=31b7342ecf07408cab6bb040f361fa1f, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:15.758875 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e670f16-913b-4af5-a4f3-c631f368817b admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:15.775393 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 830/1648] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:15 2022] PUT /identity/v3/projects/f48276ac51dd4bb1912fc670433cf6c4/users/31b7342ecf07408cab6bb040f361fa1f/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:15.776346 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e670f16-913b-4af5-a4f3-c631f368817b admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:15.778363 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-9e670f16-913b-4af5-a4f3-c631f368817b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:15.798565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:15.816283 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1365e65c-4455-4636-a768-22882ea4eeb7 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:15.839581 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e670f16-913b-4af5-a4f3-c631f368817b admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:15.843888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1aa152b-6652-44d2-a5a5-cfb57da9ad53 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:15.844883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aa152b-6652-44d2-a5a5-cfb57da9ad53 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:15.845202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aa152b-6652-44d2-a5a5-cfb57da9ad53 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:15.845479 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aa152b-6652-44d2-a5a5-cfb57da9ad53 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:15.868022 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1338db-a90d-4bd4-8ae4-67f4111bce5e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:15.868826 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1338db-a90d-4bd4-8ae4-67f4111bce5e service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:15.869169 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1338db-a90d-4bd4-8ae4-67f4111bce5e service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:15.869502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1338db-a90d-4bd4-8ae4-67f4111bce5e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:15.920052 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1aa152b-6652-44d2-a5a5-cfb57da9ad53 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:15.921613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1aa152b-6652-44d2-a5a5-cfb57da9ad53 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:15.931431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 831/1649] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:15.938216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4899a3-c63e-4eea-aac1-3781c85e21a6 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:15.939073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4899a3-c63e-4eea-aac1-3781c85e21a6 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:15.939685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4899a3-c63e-4eea-aac1-3781c85e21a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:15.947936 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1338db-a90d-4bd4-8ae4-67f4111bce5e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:15.949641 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1338db-a90d-4bd4-8ae4-67f4111bce5e service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:15.961266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 819/1650] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:15.966045 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-ef4899a3-c63e-4eea-aac1-3781c85e21a6 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:15.967251 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ef4899a3-c63e-4eea-aac1-3781c85e21a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:15.979695 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 832/1651] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:15.989230 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1338db-a90d-4bd4-8ae4-67f4111bce5e service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:15.990826 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-ce1338db-a90d-4bd4-8ae4-67f4111bce5e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:16.006250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1338db-a90d-4bd4-8ae4-67f4111bce5e service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:16.011285 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103e1a38-e128-4a70-bec9-c2d6fd537008 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:16.012252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103e1a38-e128-4a70-bec9-c2d6fd537008 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:16.012546 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103e1a38-e128-4a70-bec9-c2d6fd537008 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:16.012834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103e1a38-e128-4a70-bec9-c2d6fd537008 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:16.028902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-103e1a38-e128-4a70-bec9-c2d6fd537008 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:16.031078 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-103e1a38-e128-4a70-bec9-c2d6fd537008 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:16.042890 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 820/1652] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 00:58:16.428466 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4899a3-c63e-4eea-aac1-3781c85e21a6 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:16.430892 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ef4899a3-c63e-4eea-aac1-3781c85e21a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:16.446034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4899a3-c63e-4eea-aac1-3781c85e21a6 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:16.451862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a58962d-e13d-479c-a328-7df37dd7bf10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:16.452415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a58962d-e13d-479c-a328-7df37dd7bf10 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:16.452743 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a58962d-e13d-479c-a328-7df37dd7bf10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:16.453058 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a58962d-e13d-479c-a328-7df37dd7bf10 admin admin] PATH_INFO: `/v3/projects/776d07aafb084e2e9517aedc639f012a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:16.454190 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a58962d-e13d-479c-a328-7df37dd7bf10 admin admin] RBAC: Authorizing `identity:delete_project(project_id=776d07aafb084e2e9517aedc639f012a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:16.456118 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a58962d-e13d-479c-a328-7df37dd7bf10 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:16.547782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7a58962d-e13d-479c-a328-7df37dd7bf10 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '776d07aafb084e2e9517aedc639f012a', 'request_id': 'req-7a58962d-e13d-479c-a328-7df37dd7bf10'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:16.560505 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7a58962d-e13d-479c-a328-7df37dd7bf10 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '776d07aafb084e2e9517aedc639f012a', 'request_id': 'req-7a58962d-e13d-479c-a328-7df37dd7bf10'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:16.569558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:16.573463 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7a58962d-e13d-479c-a328-7df37dd7bf10 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '776d07aafb084e2e9517aedc639f012a', 'request_id': 'req-7a58962d-e13d-479c-a328-7df37dd7bf10'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:16.594999 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7a58962d-e13d-479c-a328-7df37dd7bf10 admin admin] The token cache is being invalidate because project 776d07aafb084e2e9517aedc639f012a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:58:16.596086 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7a58962d-e13d-479c-a328-7df37dd7bf10 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:16.597174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7a58962d-e13d-479c-a328-7df37dd7bf10 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:16.601733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 833/1653] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:58:16 2022] DELETE /identity/v3/projects/776d07aafb084e2e9517aedc639f012a => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:16.605087 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103e1a38-e128-4a70-bec9-c2d6fd537008 service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:16.620509 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103e1a38-e128-4a70-bec9-c2d6fd537008 service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:16.743557 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78bc25c-192c-4ef9-9c98-9192c4ac1b8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:16.744707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78bc25c-192c-4ef9-9c98-9192c4ac1b8d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:16.745382 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78bc25c-192c-4ef9-9c98-9192c4ac1b8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:16.745955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78bc25c-192c-4ef9-9c98-9192c4ac1b8d admin admin] PATH_INFO: `/v3/users/ec15a942520144e9917c275c5272e0c1` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:16.747373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78bc25c-192c-4ef9-9c98-9192c4ac1b8d admin admin] RBAC: Authorizing `identity:delete_user(user_id=ec15a942520144e9917c275c5272e0c1)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:16.750038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78bc25c-192c-4ef9-9c98-9192c4ac1b8d admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:16.799939 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:16.820456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-d78bc25c-192c-4ef9-9c98-9192c4ac1b8d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ec15a942520144e9917c275c5272e0c1'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:16.835418 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-d78bc25c-192c-4ef9-9c98-9192c4ac1b8d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ec15a942520144e9917c275c5272e0c1'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:16.836310 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-d78bc25c-192c-4ef9-9c98-9192c4ac1b8d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ec15a942520144e9917c275c5272e0c1'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:16.847943 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-d78bc25c-192c-4ef9-9c98-9192c4ac1b8d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ec15a942520144e9917c275c5272e0c1'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:16.867176 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-d78bc25c-192c-4ef9-9c98-9192c4ac1b8d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ec15a942520144e9917c275c5272e0c1'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:16.878578 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 821/1654] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:58:16 2022] DELETE /identity/v3/users/ec15a942520144e9917c275c5272e0c1 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:17.068200 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0b6ffc-5363-4536-bba3-e873e88db3bc None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:17.068770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0b6ffc-5363-4536-bba3-e873e88db3bc None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:17.068770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0b6ffc-5363-4536-bba3-e873e88db3bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:17.107659 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-ff0b6ffc-5363-4536-bba3-e873e88db3bc None None] MFA Rules not processed for user `ddeaa7a90ee7483590e6bc0d189e1e2a`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:17.116898 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ff0b6ffc-5363-4536-bba3-e873e88db3bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:17.254446 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 834/1655] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:17 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:17.266347 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78bc25c-192c-4ef9-9c98-9192c4ac1b8d admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:17.268233 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d78bc25c-192c-4ef9-9c98-9192c4ac1b8d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:17.289899 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0b6ffc-5363-4536-bba3-e873e88db3bc None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:17.291434 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ff0b6ffc-5363-4536-bba3-e873e88db3bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:17.382771 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78bc25c-192c-4ef9-9c98-9192c4ac1b8d admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:17.389141 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8620636-bd4b-4418-9ecb-d9511fbe16e2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:17.389766 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8620636-bd4b-4418-9ecb-d9511fbe16e2 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:17.390190 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8620636-bd4b-4418-9ecb-d9511fbe16e2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:17.390532 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8620636-bd4b-4418-9ecb-d9511fbe16e2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:17.402531 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8620636-bd4b-4418-9ecb-d9511fbe16e2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:17.403339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0b6ffc-5363-4536-bba3-e873e88db3bc None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:17.404655 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8620636-bd4b-4418-9ecb-d9511fbe16e2 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:17.409513 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a186f757-d0b2-437d-a156-ec255b6756ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:17.410384 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a186f757-d0b2-437d-a156-ec255b6756ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:17.410965 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a186f757-d0b2-437d-a156-ec255b6756ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:17.411391 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a186f757-d0b2-437d-a156-ec255b6756ac admin admin] PATH_INFO: `/v3/projects/3449cc37b0c04f3c8414acc1b9a386e7` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:17.413998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a186f757-d0b2-437d-a156-ec255b6756ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=3449cc37b0c04f3c8414acc1b9a386e7)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:17.417199 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a186f757-d0b2-437d-a156-ec255b6756ac admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:17.424015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 822/1656] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:17 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:17.498094 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a186f757-d0b2-437d-a156-ec255b6756ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3449cc37b0c04f3c8414acc1b9a386e7', 'request_id': 'req-a186f757-d0b2-437d-a156-ec255b6756ac'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:17.510274 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a186f757-d0b2-437d-a156-ec255b6756ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3449cc37b0c04f3c8414acc1b9a386e7', 'request_id': 'req-a186f757-d0b2-437d-a156-ec255b6756ac'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:17.520878 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a186f757-d0b2-437d-a156-ec255b6756ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3449cc37b0c04f3c8414acc1b9a386e7', 'request_id': 'req-a186f757-d0b2-437d-a156-ec255b6756ac'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:17.540992 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a186f757-d0b2-437d-a156-ec255b6756ac admin admin] The token cache is being invalidate because project 3449cc37b0c04f3c8414acc1b9a386e7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:58:17.541298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a186f757-d0b2-437d-a156-ec255b6756ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:17.542230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a186f757-d0b2-437d-a156-ec255b6756ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:17.543821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 835/1657] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:58:17 2022] DELETE /identity/v3/projects/3449cc37b0c04f3c8414acc1b9a386e7 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:17.549585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6057a0c3-c5ce-4ee6-850c-4b4ca0af4916 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:17.550000 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6057a0c3-c5ce-4ee6-850c-4b4ca0af4916 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:17.550449 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6057a0c3-c5ce-4ee6-850c-4b4ca0af4916 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:17.581393 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-6057a0c3-c5ce-4ee6-850c-4b4ca0af4916 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:17.584684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-6057a0c3-c5ce-4ee6-850c-4b4ca0af4916 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:17.658880 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 823/1658] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:17.667340 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b01c179-ec99-408d-976f-b8af08519518 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:17.667619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b01c179-ec99-408d-976f-b8af08519518 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:17.667914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b01c179-ec99-408d-976f-b8af08519518 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:17.690479 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-6b01c179-ec99-408d-976f-b8af08519518 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:17.692277 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-6b01c179-ec99-408d-976f-b8af08519518 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:17.701456 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 836/1659] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:17.707257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6057a0c3-c5ce-4ee6-850c-4b4ca0af4916 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:17.718393 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6057a0c3-c5ce-4ee6-850c-4b4ca0af4916 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:17.723813 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5d47bc-e122-4984-832c-9afd69c546fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:17.724625 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5d47bc-e122-4984-832c-9afd69c546fe admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:17.724960 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5d47bc-e122-4984-832c-9afd69c546fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:17.725295 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5d47bc-e122-4984-832c-9afd69c546fe admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:17.726159 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5d47bc-e122-4984-832c-9afd69c546fe admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:58:17.726524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5d47bc-e122-4984-832c-9afd69c546fe admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:17.728340 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5d47bc-e122-4984-832c-9afd69c546fe admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:17.746386 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 824/1660] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:58:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:58:17.747978 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b01c179-ec99-408d-976f-b8af08519518 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:17.753970 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:17.768134 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b01c179-ec99-408d-976f-b8af08519518 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:17.793652 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bf1b08-e4e5-4a2f-bdaf-d7004d146336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:17.794353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bf1b08-e4e5-4a2f-bdaf-d7004d146336 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:17.795057 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bf1b08-e4e5-4a2f-bdaf-d7004d146336 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:17.795395 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bf1b08-e4e5-4a2f-bdaf-d7004d146336 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:17.797532 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0bf1b08-e4e5-4a2f-bdaf-d7004d146336 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:17.804252 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0bf1b08-e4e5-4a2f-bdaf-d7004d146336 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:17.855281 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 837/1661] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:58:17 2022] POST /identity/v3/projects => generated 387 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:17.860921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5d47bc-e122-4984-832c-9afd69c546fe admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:17.871845 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5d47bc-e122-4984-832c-9afd69c546fe admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:17.913704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:17.941144 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db41a9df-2192-42fd-9d15-37d65e38a8ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:17.942198 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db41a9df-2192-42fd-9d15-37d65e38a8ea admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:17.942568 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db41a9df-2192-42fd-9d15-37d65e38a8ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:17.942938 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db41a9df-2192-42fd-9d15-37d65e38a8ea admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:17.943686 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db41a9df-2192-42fd-9d15-37d65e38a8ea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:17.946257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db41a9df-2192-42fd-9d15-37d65e38a8ea admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:17.975794 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 825/1662] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:58:17 2022] POST /identity/v3/users => generated 294 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:58:17.979505 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bf1b08-e4e5-4a2f-bdaf-d7004d146336 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:17.998768 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bf1b08-e4e5-4a2f-bdaf-d7004d146336 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:18.004257 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7c7e09-30a0-4e1d-8773-67b968a4ecf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:18.004880 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7c7e09-30a0-4e1d-8773-67b968a4ecf8 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:18.005153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7c7e09-30a0-4e1d-8773-67b968a4ecf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:18.005422 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7c7e09-30a0-4e1d-8773-67b968a4ecf8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:18.006071 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7c7e09-30a0-4e1d-8773-67b968a4ecf8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:18.007661 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7c7e09-30a0-4e1d-8773-67b968a4ecf8 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:18.023544 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 838/1663] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:17 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:18.024861 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db41a9df-2192-42fd-9d15-37d65e38a8ea admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:18.037753 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db41a9df-2192-42fd-9d15-37d65e38a8ea admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:18.044388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81a5782-122f-47a9-b951-ef76b42301fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:18.045489 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81a5782-122f-47a9-b951-ef76b42301fd admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:18.045942 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81a5782-122f-47a9-b951-ef76b42301fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:18.046341 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81a5782-122f-47a9-b951-ef76b42301fd admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:18.047192 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81a5782-122f-47a9-b951-ef76b42301fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:18.049476 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81a5782-122f-47a9-b951-ef76b42301fd admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:18.062912 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 826/1664] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:18 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:18.067301 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7c7e09-30a0-4e1d-8773-67b968a4ecf8 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:18.079921 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7c7e09-30a0-4e1d-8773-67b968a4ecf8 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:18.085342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bb2d08-0d71-40bc-8fd8-8717e2bae4f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:18.086009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bb2d08-0d71-40bc-8fd8-8717e2bae4f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:18.086283 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bb2d08-0d71-40bc-8fd8-8717e2bae4f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:18.086605 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bb2d08-0d71-40bc-8fd8-8717e2bae4f4 admin admin] PATH_INFO: `/v3/projects/482a813987c24647850c4de94a1e1b54/users/fee8aa7766dc449a9be7eb733f980de7/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:18.104558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bb2d08-0d71-40bc-8fd8-8717e2bae4f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=482a813987c24647850c4de94a1e1b54, user_id=fee8aa7766dc449a9be7eb733f980de7, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:18.107480 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bb2d08-0d71-40bc-8fd8-8717e2bae4f4 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:18.124103 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 839/1665] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:18 2022] PUT /identity/v3/projects/482a813987c24647850c4de94a1e1b54/users/fee8aa7766dc449a9be7eb733f980de7/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:18.137915 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e75ee41-eca7-458c-9e5c-e05fefc88523 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:18.138121 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e75ee41-eca7-458c-9e5c-e05fefc88523 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:18.138871 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e75ee41-eca7-458c-9e5c-e05fefc88523 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:18.183105 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-0e75ee41-eca7-458c-9e5c-e05fefc88523 None None] MFA Rules not processed for user `fee8aa7766dc449a9be7eb733f980de7`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:18.184353 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-0e75ee41-eca7-458c-9e5c-e05fefc88523 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:18.297786 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 827/1666] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:18 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:18.302566 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bb2d08-0d71-40bc-8fd8-8717e2bae4f4 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:18.377494 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bb2d08-0d71-40bc-8fd8-8717e2bae4f4 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:18.397139 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1254cb75-0c1f-4889-9c55-016595ed55fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:18.399372 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1254cb75-0c1f-4889-9c55-016595ed55fd admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:18.399372 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1254cb75-0c1f-4889-9c55-016595ed55fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:18.399372 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1254cb75-0c1f-4889-9c55-016595ed55fd admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:18.401515 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1254cb75-0c1f-4889-9c55-016595ed55fd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:18.403600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1254cb75-0c1f-4889-9c55-016595ed55fd admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:18.450688 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 840/1667] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:58:18 2022] POST /identity/v3/projects => generated 387 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:58:18.455278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e75ee41-eca7-458c-9e5c-e05fefc88523 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:18.467090 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e75ee41-eca7-458c-9e5c-e05fefc88523 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:18.533631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0306f32-36bb-4ac9-baa3-1f4f8ab09121 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:18.534250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0306f32-36bb-4ac9-baa3-1f4f8ab09121 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:18.534525 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0306f32-36bb-4ac9-baa3-1f4f8ab09121 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:18.534798 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0306f32-36bb-4ac9-baa3-1f4f8ab09121 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:18.535430 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0306f32-36bb-4ac9-baa3-1f4f8ab09121 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:18.537631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0306f32-36bb-4ac9-baa3-1f4f8ab09121 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:18.559294 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 828/1668] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:58:18 2022] POST /identity/v3/users => generated 300 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:18.563587 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1254cb75-0c1f-4889-9c55-016595ed55fd admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:18.574922 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1254cb75-0c1f-4889-9c55-016595ed55fd admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:18.579894 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5c1ec2-6148-4824-96f9-40cb84e4cec9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:18.580505 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5c1ec2-6148-4824-96f9-40cb84e4cec9 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:18.580780 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5c1ec2-6148-4824-96f9-40cb84e4cec9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:18.581046 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5c1ec2-6148-4824-96f9-40cb84e4cec9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:18.581816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5c1ec2-6148-4824-96f9-40cb84e4cec9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:18.583448 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5c1ec2-6148-4824-96f9-40cb84e4cec9 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:18.595675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 841/1669] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:18 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:18.600210 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0306f32-36bb-4ac9-baa3-1f4f8ab09121 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:18.610334 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0306f32-36bb-4ac9-baa3-1f4f8ab09121 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:18.615094 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53d8a63-ea81-4193-94c1-931134b05df7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:18.615782 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53d8a63-ea81-4193-94c1-931134b05df7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:18.616084 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53d8a63-ea81-4193-94c1-931134b05df7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:18.616353 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53d8a63-ea81-4193-94c1-931134b05df7 admin admin] PATH_INFO: `/v3/projects/44b17e92302640d68f8539120a126f16/users/6d781226bda74f978e85060bd84ced30/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:18.631320 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53d8a63-ea81-4193-94c1-931134b05df7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=44b17e92302640d68f8539120a126f16, user_id=6d781226bda74f978e85060bd84ced30, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:18.633990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53d8a63-ea81-4193-94c1-931134b05df7 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:18.649885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 829/1670] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:18 2022] PUT /identity/v3/projects/44b17e92302640d68f8539120a126f16/users/6d781226bda74f978e85060bd84ced30/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:18.654223 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5c1ec2-6148-4824-96f9-40cb84e4cec9 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:18.668608 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5c1ec2-6148-4824-96f9-40cb84e4cec9 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:18.730608 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bccd43ae-9f29-4d9e-a4fc-00a8085d7279 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:18.731546 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccd43ae-9f29-4d9e-a4fc-00a8085d7279 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:18.731856 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccd43ae-9f29-4d9e-a4fc-00a8085d7279 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:18.732129 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccd43ae-9f29-4d9e-a4fc-00a8085d7279 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:18.732763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bccd43ae-9f29-4d9e-a4fc-00a8085d7279 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:18.734601 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bccd43ae-9f29-4d9e-a4fc-00a8085d7279 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:18.745635 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 842/1671] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:18 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:18.749579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53d8a63-ea81-4193-94c1-931134b05df7 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:18.760002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53d8a63-ea81-4193-94c1-931134b05df7 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:18.764886 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3ad744-bcad-4103-b040-10b29bae033e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:18.765653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3ad744-bcad-4103-b040-10b29bae033e admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:18.765917 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3ad744-bcad-4103-b040-10b29bae033e admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:18.766180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3ad744-bcad-4103-b040-10b29bae033e admin admin] PATH_INFO: `/v3/projects/44b17e92302640d68f8539120a126f16/users/6d781226bda74f978e85060bd84ced30/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:18.768162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3ad744-bcad-4103-b040-10b29bae033e admin admin] RBAC: Authorizing `identity:create_grant(project_id=44b17e92302640d68f8539120a126f16, user_id=6d781226bda74f978e85060bd84ced30, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:18.770774 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3ad744-bcad-4103-b040-10b29bae033e admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:18.777237 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:18.782223 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 830/1672] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:18 2022] PUT /identity/v3/projects/44b17e92302640d68f8539120a126f16/users/6d781226bda74f978e85060bd84ced30/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:18.792851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7ad4c4-c9a7-4a0b-ab7d-ace776579480 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:18.793135 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7ad4c4-c9a7-4a0b-ab7d-ace776579480 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:18.793412 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7ad4c4-c9a7-4a0b-ab7d-ace776579480 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:18.826136 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-bc7ad4c4-c9a7-4a0b-ab7d-ace776579480 None None] MFA Rules not processed for user `6d781226bda74f978e85060bd84ced30`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:18.827693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-bc7ad4c4-c9a7-4a0b-ab7d-ace776579480 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:18.831974 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:18.932017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 843/1673] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:18 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:18.940688 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3ad744-bcad-4103-b040-10b29bae033e admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:18.942609 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-9d3ad744-bcad-4103-b040-10b29bae033e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:19.019691 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3ad744-bcad-4103-b040-10b29bae033e admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:19.025326 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5448241-3774-4585-8c47-1fe2aa860a17 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:19.025961 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5448241-3774-4585-8c47-1fe2aa860a17 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:19.026236 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5448241-3774-4585-8c47-1fe2aa860a17 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:19.026507 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5448241-3774-4585-8c47-1fe2aa860a17 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:19.037418 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5448241-3774-4585-8c47-1fe2aa860a17 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:19.038881 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5448241-3774-4585-8c47-1fe2aa860a17 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:19.051304 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 831/1674] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:18 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:19.137714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7ad4c4-c9a7-4a0b-ab7d-ace776579480 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:19.239310 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7ad4c4-c9a7-4a0b-ab7d-ace776579480 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:19.244673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b809d32c-f3e7-429f-a8c1-99cf05b34d3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:19.247644 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b809d32c-f3e7-429f-a8c1-99cf05b34d3a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:19.247981 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b809d32c-f3e7-429f-a8c1-99cf05b34d3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:19.248357 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b809d32c-f3e7-429f-a8c1-99cf05b34d3a admin admin] PATH_INFO: `/v3/users/fee8aa7766dc449a9be7eb733f980de7` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:19.249887 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b809d32c-f3e7-429f-a8c1-99cf05b34d3a admin admin] RBAC: Authorizing `identity:delete_user(user_id=fee8aa7766dc449a9be7eb733f980de7)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:19.252339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b809d32c-f3e7-429f-a8c1-99cf05b34d3a admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:19.318609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b809d32c-f3e7-429f-a8c1-99cf05b34d3a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fee8aa7766dc449a9be7eb733f980de7'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:19.330000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b809d32c-f3e7-429f-a8c1-99cf05b34d3a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fee8aa7766dc449a9be7eb733f980de7'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:19.330684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b809d32c-f3e7-429f-a8c1-99cf05b34d3a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fee8aa7766dc449a9be7eb733f980de7'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:19.356239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b809d32c-f3e7-429f-a8c1-99cf05b34d3a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fee8aa7766dc449a9be7eb733f980de7'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:19.372803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b809d32c-f3e7-429f-a8c1-99cf05b34d3a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fee8aa7766dc449a9be7eb733f980de7'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:19.381084 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 844/1675] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:58:19 2022] DELETE /identity/v3/users/fee8aa7766dc449a9be7eb733f980de7 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:19.385715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3202fc50-7fe2-401f-b395-2de5caeb4c17 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:19.386802 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3202fc50-7fe2-401f-b395-2de5caeb4c17 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:19.387612 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3202fc50-7fe2-401f-b395-2de5caeb4c17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:19.408267 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-3202fc50-7fe2-401f-b395-2de5caeb4c17 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:19.409408 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-3202fc50-7fe2-401f-b395-2de5caeb4c17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:19.479562 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 832/1676] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:58:19.490736 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b809d32c-f3e7-429f-a8c1-99cf05b34d3a admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:19.493143 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-b809d32c-f3e7-429f-a8c1-99cf05b34d3a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:19.519105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:19.569528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b809d32c-f3e7-429f-a8c1-99cf05b34d3a admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:19.573950 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250c2e08-beea-4c48-a0b8-343c3469ebe4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:19.574577 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250c2e08-beea-4c48-a0b8-343c3469ebe4 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:19.574855 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250c2e08-beea-4c48-a0b8-343c3469ebe4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:19.575554 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250c2e08-beea-4c48-a0b8-343c3469ebe4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:19.589540 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250c2e08-beea-4c48-a0b8-343c3469ebe4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:19.591061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250c2e08-beea-4c48-a0b8-343c3469ebe4 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:19.603437 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 845/1677] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:19.992938 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3202fc50-7fe2-401f-b395-2de5caeb4c17 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:19.994487 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-3202fc50-7fe2-401f-b395-2de5caeb4c17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:20.014480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3202fc50-7fe2-401f-b395-2de5caeb4c17 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:20.026070 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458f5bc8-2f1d-491f-9750-21a9b339da8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:20.026864 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458f5bc8-2f1d-491f-9750-21a9b339da8a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:20.027306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458f5bc8-2f1d-491f-9750-21a9b339da8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:20.027775 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458f5bc8-2f1d-491f-9750-21a9b339da8a admin admin] PATH_INFO: `/v3/projects/482a813987c24647850c4de94a1e1b54` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:20.029808 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458f5bc8-2f1d-491f-9750-21a9b339da8a admin admin] RBAC: Authorizing `identity:delete_project(project_id=482a813987c24647850c4de94a1e1b54)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:20.032963 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458f5bc8-2f1d-491f-9750-21a9b339da8a admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:20.112752 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-458f5bc8-2f1d-491f-9750-21a9b339da8a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '482a813987c24647850c4de94a1e1b54', 'request_id': 'req-458f5bc8-2f1d-491f-9750-21a9b339da8a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:20.134583 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-458f5bc8-2f1d-491f-9750-21a9b339da8a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '482a813987c24647850c4de94a1e1b54', 'request_id': 'req-458f5bc8-2f1d-491f-9750-21a9b339da8a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:20.152236 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-458f5bc8-2f1d-491f-9750-21a9b339da8a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '482a813987c24647850c4de94a1e1b54', 'request_id': 'req-458f5bc8-2f1d-491f-9750-21a9b339da8a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:20.168399 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-458f5bc8-2f1d-491f-9750-21a9b339da8a admin admin] The token cache is being invalidate because project 482a813987c24647850c4de94a1e1b54 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:58:20.168944 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-458f5bc8-2f1d-491f-9750-21a9b339da8a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:20.169732 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-458f5bc8-2f1d-491f-9750-21a9b339da8a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:20.172151 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 833/1678] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:58:19 2022] DELETE /identity/v3/projects/482a813987c24647850c4de94a1e1b54 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:20.176116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250c2e08-beea-4c48-a0b8-343c3469ebe4 service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:20.191100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250c2e08-beea-4c48-a0b8-343c3469ebe4 service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:20.298159 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c614f995-9e3b-4011-8760-c8ab1d20b748 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:20.298802 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c614f995-9e3b-4011-8760-c8ab1d20b748 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:20.299040 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c614f995-9e3b-4011-8760-c8ab1d20b748 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:20.299315 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c614f995-9e3b-4011-8760-c8ab1d20b748 admin admin] PATH_INFO: `/v3/users/6d781226bda74f978e85060bd84ced30` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:20.300542 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c614f995-9e3b-4011-8760-c8ab1d20b748 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6d781226bda74f978e85060bd84ced30)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:20.303047 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c614f995-9e3b-4011-8760-c8ab1d20b748 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:20.393044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c614f995-9e3b-4011-8760-c8ab1d20b748 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6d781226bda74f978e85060bd84ced30'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:20.407185 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c614f995-9e3b-4011-8760-c8ab1d20b748 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6d781226bda74f978e85060bd84ced30'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:20.408309 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c614f995-9e3b-4011-8760-c8ab1d20b748 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6d781226bda74f978e85060bd84ced30'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:20.428793 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c614f995-9e3b-4011-8760-c8ab1d20b748 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6d781226bda74f978e85060bd84ced30'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:20.442038 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c614f995-9e3b-4011-8760-c8ab1d20b748 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6d781226bda74f978e85060bd84ced30'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:20.460744 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 846/1679] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:58:20 2022] DELETE /identity/v3/users/6d781226bda74f978e85060bd84ced30 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:20.855406 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458f5bc8-2f1d-491f-9750-21a9b339da8a admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:20.857212 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-458f5bc8-2f1d-491f-9750-21a9b339da8a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:20.880239 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458f5bc8-2f1d-491f-9750-21a9b339da8a admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:20.918069 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:20.959779 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e759f2c-3fe7-43ad-8cbe-12afda263fb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:20.960438 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e759f2c-3fe7-43ad-8cbe-12afda263fb6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:20.960714 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e759f2c-3fe7-43ad-8cbe-12afda263fb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:20.960984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e759f2c-3fe7-43ad-8cbe-12afda263fb6 admin admin] PATH_INFO: `/v3/projects/44b17e92302640d68f8539120a126f16` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:20.962337 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e759f2c-3fe7-43ad-8cbe-12afda263fb6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=44b17e92302640d68f8539120a126f16)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:20.964349 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e759f2c-3fe7-43ad-8cbe-12afda263fb6 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:20.990371 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3645533-94ba-4212-badf-5eba5717cb49 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:20.990660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3645533-94ba-4212-badf-5eba5717cb49 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:20.990934 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3645533-94ba-4212-badf-5eba5717cb49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:21.023239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-f3645533-94ba-4212-badf-5eba5717cb49 None None] MFA Rules not processed for user `31b7342ecf07408cab6bb040f361fa1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:21.024544 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-f3645533-94ba-4212-badf-5eba5717cb49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:21.035064 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4e759f2c-3fe7-43ad-8cbe-12afda263fb6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '44b17e92302640d68f8539120a126f16', 'request_id': 'req-4e759f2c-3fe7-43ad-8cbe-12afda263fb6'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:21.044212 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4e759f2c-3fe7-43ad-8cbe-12afda263fb6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '44b17e92302640d68f8539120a126f16', 'request_id': 'req-4e759f2c-3fe7-43ad-8cbe-12afda263fb6'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:21.057032 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4e759f2c-3fe7-43ad-8cbe-12afda263fb6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '44b17e92302640d68f8539120a126f16', 'request_id': 'req-4e759f2c-3fe7-43ad-8cbe-12afda263fb6'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:21.065620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:21.074314 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4e759f2c-3fe7-43ad-8cbe-12afda263fb6 admin admin] The token cache is being invalidate because project 44b17e92302640d68f8539120a126f16 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:58:21.074801 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4e759f2c-3fe7-43ad-8cbe-12afda263fb6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:21.075807 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4e759f2c-3fe7-43ad-8cbe-12afda263fb6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:21.077920 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 834/1680] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:58:20 2022] DELETE /identity/v3/projects/44b17e92302640d68f8539120a126f16 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:21.085556 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d380df29-4c5c-4ffd-aa6b-02660a015ec6 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:21.085909 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d380df29-4c5c-4ffd-aa6b-02660a015ec6 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:21.086100 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d380df29-4c5c-4ffd-aa6b-02660a015ec6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:21.122845 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 847/1681] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:20 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:58:21.131945 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-d380df29-4c5c-4ffd-aa6b-02660a015ec6 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:21.136673 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d380df29-4c5c-4ffd-aa6b-02660a015ec6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:21.139067 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b02ea3-6575-4299-8160-97ec5596393e None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:21.139545 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b02ea3-6575-4299-8160-97ec5596393e None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:21.140087 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b02ea3-6575-4299-8160-97ec5596393e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:21.163319 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-c1b02ea3-6575-4299-8160-97ec5596393e None None] MFA Rules not processed for user `31b7342ecf07408cab6bb040f361fa1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:21.164737 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c1b02ea3-6575-4299-8160-97ec5596393e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:21.172454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 848/1682] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:21 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:21.176544 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da4b2b0-2349-45b9-bc3e-9caed2fbd730 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:21.176544 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da4b2b0-2349-45b9-bc3e-9caed2fbd730 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:21.176544 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da4b2b0-2349-45b9-bc3e-9caed2fbd730 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:21.195924 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-0da4b2b0-2349-45b9-bc3e-9caed2fbd730 None None] MFA Rules not processed for user `ddeaa7a90ee7483590e6bc0d189e1e2a`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:21.197121 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-0da4b2b0-2349-45b9-bc3e-9caed2fbd730 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:21.228164 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 835/1683] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:21.228164 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d380df29-4c5c-4ffd-aa6b-02660a015ec6 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:21.229617 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d380df29-4c5c-4ffd-aa6b-02660a015ec6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:21.314095 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 849/1684] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:21 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:21.315784 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd344da4-08c1-4ee7-aeb9-ce2a193c1bbe None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:21.316404 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd344da4-08c1-4ee7-aeb9-ce2a193c1bbe None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:21.316907 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:21.317256 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd344da4-08c1-4ee7-aeb9-ce2a193c1bbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:21.352879 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d380df29-4c5c-4ffd-aa6b-02660a015ec6 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:21.353877 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-dd344da4-08c1-4ee7-aeb9-ce2a193c1bbe None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:21.355300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-dd344da4-08c1-4ee7-aeb9-ce2a193c1bbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:21.359002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd10d3b9-3491-4027-8fbc-8b01f9f4102b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:21.360074 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd10d3b9-3491-4027-8fbc-8b01f9f4102b service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:21.360423 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd10d3b9-3491-4027-8fbc-8b01f9f4102b service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:21.360778 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd10d3b9-3491-4027-8fbc-8b01f9f4102b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:21.364958 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 850/1685] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:21.372080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd344da4-08c1-4ee7-aeb9-ce2a193c1bbe None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:21.373883 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd10d3b9-3491-4027-8fbc-8b01f9f4102b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:21.377197 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd10d3b9-3491-4027-8fbc-8b01f9f4102b service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:21.391527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd344da4-08c1-4ee7-aeb9-ce2a193c1bbe None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:21.398402 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 836/1686] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:21 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:21.404750 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84afdda3-d38d-47c4-b52d-9e29f5139be7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:21.405379 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84afdda3-d38d-47c4-b52d-9e29f5139be7 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:21.405744 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84afdda3-d38d-47c4-b52d-9e29f5139be7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:21.406604 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84afdda3-d38d-47c4-b52d-9e29f5139be7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:21.407361 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84afdda3-d38d-47c4-b52d-9e29f5139be7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:58:21.407679 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84afdda3-d38d-47c4-b52d-9e29f5139be7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:21.410506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84afdda3-d38d-47c4-b52d-9e29f5139be7 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:21.439623 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 851/1687] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:58:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:58:21.440258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd10d3b9-3491-4027-8fbc-8b01f9f4102b service nova] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:21.460360 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd10d3b9-3491-4027-8fbc-8b01f9f4102b service nova] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:21.471066 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625d0bf9-1aff-4562-8d54-1fcadb21b9fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:21.471669 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625d0bf9-1aff-4562-8d54-1fcadb21b9fa admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:21.471979 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625d0bf9-1aff-4562-8d54-1fcadb21b9fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:21.472245 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625d0bf9-1aff-4562-8d54-1fcadb21b9fa admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:21.472872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625d0bf9-1aff-4562-8d54-1fcadb21b9fa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:21.474839 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625d0bf9-1aff-4562-8d54-1fcadb21b9fa admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:21.519165 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84afdda3-d38d-47c4-b52d-9e29f5139be7 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:21.522107 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-84afdda3-d38d-47c4-b52d-9e29f5139be7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:21.528584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 837/1688] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:58:21 2022] POST /identity/v3/projects => generated 385 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:58:21.533526 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625d0bf9-1aff-4562-8d54-1fcadb21b9fa admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:21.538369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84afdda3-d38d-47c4-b52d-9e29f5139be7 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:21.620094 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a73e28-1532-4848-a0f7-a1c8b6875be5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:21.621199 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a73e28-1532-4848-a0f7-a1c8b6875be5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:21.621782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a73e28-1532-4848-a0f7-a1c8b6875be5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:21.622694 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a73e28-1532-4848-a0f7-a1c8b6875be5 admin admin] PATH_INFO: `/v3/users/44be814d35bc49888ef6aa629f6af604` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:21.623163 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625d0bf9-1aff-4562-8d54-1fcadb21b9fa admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:21.624472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a73e28-1532-4848-a0f7-a1c8b6875be5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=44be814d35bc49888ef6aa629f6af604)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:21.627592 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a73e28-1532-4848-a0f7-a1c8b6875be5 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:21.628961 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6eaf8b-9a63-4358-b731-3495d92618de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:21.629937 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6eaf8b-9a63-4358-b731-3495d92618de admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:21.642771 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6eaf8b-9a63-4358-b731-3495d92618de admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:21.643159 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6eaf8b-9a63-4358-b731-3495d92618de admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:21.644012 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6eaf8b-9a63-4358-b731-3495d92618de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:21.651324 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6eaf8b-9a63-4358-b731-3495d92618de admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:21.692674 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 838/1689] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:58:21 2022] POST /identity/v3/users => generated 293 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:21.696735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6eaf8b-9a63-4358-b731-3495d92618de admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:21.710607 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6eaf8b-9a63-4358-b731-3495d92618de admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:21.716610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674112ed-a34f-4e1d-9534-a5374e0c1c11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:21.717419 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674112ed-a34f-4e1d-9534-a5374e0c1c11 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:21.717851 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674112ed-a34f-4e1d-9534-a5374e0c1c11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:21.718656 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674112ed-a34f-4e1d-9534-a5374e0c1c11 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:21.719509 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674112ed-a34f-4e1d-9534-a5374e0c1c11 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:21.721509 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674112ed-a34f-4e1d-9534-a5374e0c1c11 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:21.729914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b4a73e28-1532-4848-a0f7-a1c8b6875be5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '44be814d35bc49888ef6aa629f6af604'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:21.736427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 839/1690] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:21 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 00:58:21.740939 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674112ed-a34f-4e1d-9534-a5374e0c1c11 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:21.756755 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674112ed-a34f-4e1d-9534-a5374e0c1c11 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:21.757600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:21.764607 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0068b44b-076f-47de-99c5-83cb8fe1e11e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:21.765392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0068b44b-076f-47de-99c5-83cb8fe1e11e admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:21.765732 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b4a73e28-1532-4848-a0f7-a1c8b6875be5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '44be814d35bc49888ef6aa629f6af604'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:21.765894 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0068b44b-076f-47de-99c5-83cb8fe1e11e admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:21.766125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0068b44b-076f-47de-99c5-83cb8fe1e11e admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:21.767170 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0068b44b-076f-47de-99c5-83cb8fe1e11e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:21.768925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0068b44b-076f-47de-99c5-83cb8fe1e11e admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:21.781237 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 840/1691] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:21 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 00:58:21.785972 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b4a73e28-1532-4848-a0f7-a1c8b6875be5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '44be814d35bc49888ef6aa629f6af604'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:21.787775 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0068b44b-076f-47de-99c5-83cb8fe1e11e admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:21.791142 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-0068b44b-076f-47de-99c5-83cb8fe1e11e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:21.811182 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b4a73e28-1532-4848-a0f7-a1c8b6875be5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '44be814d35bc49888ef6aa629f6af604'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:21.815093 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0068b44b-076f-47de-99c5-83cb8fe1e11e admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:21.826529 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b4a73e28-1532-4848-a0f7-a1c8b6875be5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '44be814d35bc49888ef6aa629f6af604'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:21.832779 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96424fe5-15e0-4a56-8fe3-f20a6ebb2b75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:21.833908 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96424fe5-15e0-4a56-8fe3-f20a6ebb2b75 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:21.834455 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96424fe5-15e0-4a56-8fe3-f20a6ebb2b75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:21.834736 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96424fe5-15e0-4a56-8fe3-f20a6ebb2b75 admin admin] PATH_INFO: `/v3/projects/2c8e4152267645dc9b8537ec5109d314/users/9df7cddf69454122a5a3f5d3dd2162cd/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:21.846659 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 852/1692] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:58:21 2022] DELETE /identity/v3/users/44be814d35bc49888ef6aa629f6af604 => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 00:58:21.848299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf89606-4adb-49c9-bf1e-c06d1e46ce66 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:21.848583 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf89606-4adb-49c9-bf1e-c06d1e46ce66 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:21.848858 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf89606-4adb-49c9-bf1e-c06d1e46ce66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:21.878227 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96424fe5-15e0-4a56-8fe3-f20a6ebb2b75 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2c8e4152267645dc9b8537ec5109d314, user_id=9df7cddf69454122a5a3f5d3dd2162cd, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:21.880974 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96424fe5-15e0-4a56-8fe3-f20a6ebb2b75 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:21.891590 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-4bf89606-4adb-49c9-bf1e-c06d1e46ce66 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:21.893325 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-4bf89606-4adb-49c9-bf1e-c06d1e46ce66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:21.906064 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 841/1693] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:21 2022] PUT /identity/v3/projects/2c8e4152267645dc9b8537ec5109d314/users/9df7cddf69454122a5a3f5d3dd2162cd/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:21.916442 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306af4f9-0696-41e7-8662-fd1b22ecc15e None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:21.916669 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306af4f9-0696-41e7-8662-fd1b22ecc15e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:21.916931 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306af4f9-0696-41e7-8662-fd1b22ecc15e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:21.982654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-306af4f9-0696-41e7-8662-fd1b22ecc15e None None] MFA Rules not processed for user `9df7cddf69454122a5a3f5d3dd2162cd`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:21.984318 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-306af4f9-0696-41e7-8662-fd1b22ecc15e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:22.010682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 853/1694] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:22.020853 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf89606-4adb-49c9-bf1e-c06d1e46ce66 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:22.022527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-4bf89606-4adb-49c9-bf1e-c06d1e46ce66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:22.079406 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:22.096976 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf89606-4adb-49c9-bf1e-c06d1e46ce66 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:22.103372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 842/1695] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:21 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:58:22.110031 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306af4f9-0696-41e7-8662-fd1b22ecc15e None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:22.162594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028ce04d-7c56-4dd6-bf86-20a0f6c6cb6a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:22.163944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028ce04d-7c56-4dd6-bf86-20a0f6c6cb6a service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:22.163944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028ce04d-7c56-4dd6-bf86-20a0f6c6cb6a service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:22.163944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028ce04d-7c56-4dd6-bf86-20a0f6c6cb6a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:22.206814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306af4f9-0696-41e7-8662-fd1b22ecc15e None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:22.212733 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dcf7999-4d69-4ae3-bfed-545fdee98157 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:22.213363 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dcf7999-4d69-4ae3-bfed-545fdee98157 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:22.213639 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dcf7999-4d69-4ae3-bfed-545fdee98157 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:22.213908 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dcf7999-4d69-4ae3-bfed-545fdee98157 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:22.214544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dcf7999-4d69-4ae3-bfed-545fdee98157 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:22.216696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dcf7999-4d69-4ae3-bfed-545fdee98157 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:22.263016 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 843/1696] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:58:22 2022] POST /identity/v3/projects => generated 385 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:22.263206 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:22.264226 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dcf7999-4d69-4ae3-bfed-545fdee98157 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:22.299745 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028ce04d-7c56-4dd6-bf86-20a0f6c6cb6a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:22.302086 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028ce04d-7c56-4dd6-bf86-20a0f6c6cb6a service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:22.317809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 854/1697] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:22.388822 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dcf7999-4d69-4ae3-bfed-545fdee98157 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:22.402199 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4ce20b-fbed-42ef-9aa2-758e12ba321b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:22.402792 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4ce20b-fbed-42ef-9aa2-758e12ba321b admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:22.402963 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4ce20b-fbed-42ef-9aa2-758e12ba321b admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:22.403218 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4ce20b-fbed-42ef-9aa2-758e12ba321b admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:22.403914 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4ce20b-fbed-42ef-9aa2-758e12ba321b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:22.405926 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4ce20b-fbed-42ef-9aa2-758e12ba321b admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:22.454953 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 844/1698] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:58:22 2022] POST /identity/v3/users => generated 293 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:58:22.459933 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028ce04d-7c56-4dd6-bf86-20a0f6c6cb6a service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:22.472978 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028ce04d-7c56-4dd6-bf86-20a0f6c6cb6a service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:22.485017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0283bb9-4134-450b-a023-1c9948ad3a07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:22.485780 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0283bb9-4134-450b-a023-1c9948ad3a07 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:22.489407 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0283bb9-4134-450b-a023-1c9948ad3a07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:22.489937 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0283bb9-4134-450b-a023-1c9948ad3a07 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:22.490821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0283bb9-4134-450b-a023-1c9948ad3a07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:22.493173 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0283bb9-4134-450b-a023-1c9948ad3a07 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:22.510732 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 855/1699] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:22 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:22.513559 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4ce20b-fbed-42ef-9aa2-758e12ba321b admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:22.530003 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4ce20b-fbed-42ef-9aa2-758e12ba321b admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:22.540201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1341e28e-bb50-4ff5-8c9e-0c5c20f4fb3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:22.541484 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1341e28e-bb50-4ff5-8c9e-0c5c20f4fb3f admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:22.541832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1341e28e-bb50-4ff5-8c9e-0c5c20f4fb3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:22.542161 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1341e28e-bb50-4ff5-8c9e-0c5c20f4fb3f admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:22.542994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1341e28e-bb50-4ff5-8c9e-0c5c20f4fb3f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:22.545432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1341e28e-bb50-4ff5-8c9e-0c5c20f4fb3f admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:22.564383 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 845/1700] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:22 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:22.565150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0283bb9-4134-450b-a023-1c9948ad3a07 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:22.584134 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0283bb9-4134-450b-a023-1c9948ad3a07 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:22.630678 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b26f06d-a171-4a63-832a-eb56b5f3688e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:22.631392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b26f06d-a171-4a63-832a-eb56b5f3688e admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:22.631674 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b26f06d-a171-4a63-832a-eb56b5f3688e admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:22.631978 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b26f06d-a171-4a63-832a-eb56b5f3688e admin admin] PATH_INFO: `/v3/projects/0c52a0b9b1704717b7d6c6ab88947b6b/users/fb0ba64e4211465ea31c12abe6470168/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:22.655176 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b26f06d-a171-4a63-832a-eb56b5f3688e admin admin] RBAC: Authorizing `identity:create_grant(project_id=0c52a0b9b1704717b7d6c6ab88947b6b, user_id=fb0ba64e4211465ea31c12abe6470168, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:22.658889 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b26f06d-a171-4a63-832a-eb56b5f3688e admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:22.689250 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 856/1701] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:22 2022] PUT /identity/v3/projects/0c52a0b9b1704717b7d6c6ab88947b6b/users/fb0ba64e4211465ea31c12abe6470168/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:22.691551 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e115e87-60ce-4ee2-bfc0-93def8321cd6 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:22.691930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e115e87-60ce-4ee2-bfc0-93def8321cd6 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:22.692266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e115e87-60ce-4ee2-bfc0-93def8321cd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:22.698601 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887bf8c5-7f68-4ed4-8de7-73d66c1dad49 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:22.699079 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887bf8c5-7f68-4ed4-8de7-73d66c1dad49 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:22.699470 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887bf8c5-7f68-4ed4-8de7-73d66c1dad49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:22.754789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-887bf8c5-7f68-4ed4-8de7-73d66c1dad49 None None] MFA Rules not processed for user `31b7342ecf07408cab6bb040f361fa1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:22.755742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-887bf8c5-7f68-4ed4-8de7-73d66c1dad49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:22.827279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-4e115e87-60ce-4ee2-bfc0-93def8321cd6 None None] MFA Rules not processed for user `fb0ba64e4211465ea31c12abe6470168`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:22.829012 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-4e115e87-60ce-4ee2-bfc0-93def8321cd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:22.889326 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 857/1702] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:22 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:22.889326 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887bf8c5-7f68-4ed4-8de7-73d66c1dad49 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:22.889326 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-887bf8c5-7f68-4ed4-8de7-73d66c1dad49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:22.950917 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:22.976453 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887bf8c5-7f68-4ed4-8de7-73d66c1dad49 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:22.982193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772e29ab-3d28-41ce-88bc-b2a21f70a536 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:22.983023 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772e29ab-3d28-41ce-88bc-b2a21f70a536 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:22.983432 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772e29ab-3d28-41ce-88bc-b2a21f70a536 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:22.984013 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772e29ab-3d28-41ce-88bc-b2a21f70a536 admin admin] PATH_INFO: `/v3/projects/087b66d74f7a4b67820b8d0872eaa280` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:22.985314 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772e29ab-3d28-41ce-88bc-b2a21f70a536 admin admin] RBAC: Authorizing `identity:delete_project(project_id=087b66d74f7a4b67820b8d0872eaa280)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:22.987873 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772e29ab-3d28-41ce-88bc-b2a21f70a536 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:23.027684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 846/1703] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:22 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 337 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:23.033774 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e115e87-60ce-4ee2-bfc0-93def8321cd6 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:23.052009 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e115e87-60ce-4ee2-bfc0-93def8321cd6 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:23.058046 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3eab3c-beba-49da-b0c4-d51a9d854e74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:23.058758 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3eab3c-beba-49da-b0c4-d51a9d854e74 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:23.059096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3eab3c-beba-49da-b0c4-d51a9d854e74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:23.059288 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3eab3c-beba-49da-b0c4-d51a9d854e74 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:23.060889 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3eab3c-beba-49da-b0c4-d51a9d854e74 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:23.065860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3eab3c-beba-49da-b0c4-d51a9d854e74 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:23.093483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-772e29ab-3d28-41ce-88bc-b2a21f70a536 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '087b66d74f7a4b67820b8d0872eaa280', 'request_id': 'req-772e29ab-3d28-41ce-88bc-b2a21f70a536'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:23.098119 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=95,keystone:INSERT=1,keystone:DELETE=4 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:23.108449 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-772e29ab-3d28-41ce-88bc-b2a21f70a536 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '087b66d74f7a4b67820b8d0872eaa280', 'request_id': 'req-772e29ab-3d28-41ce-88bc-b2a21f70a536'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:23.117916 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 847/1704] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:58:23 2022] POST /identity/v3/projects => generated 383 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:58:23.125559 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3eab3c-beba-49da-b0c4-d51a9d854e74 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:23.129790 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-772e29ab-3d28-41ce-88bc-b2a21f70a536 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '087b66d74f7a4b67820b8d0872eaa280', 'request_id': 'req-772e29ab-3d28-41ce-88bc-b2a21f70a536'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:23.151517 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-772e29ab-3d28-41ce-88bc-b2a21f70a536 admin admin] The token cache is being invalidate because project 087b66d74f7a4b67820b8d0872eaa280 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:58:23.151955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-772e29ab-3d28-41ce-88bc-b2a21f70a536 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:23.152800 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-772e29ab-3d28-41ce-88bc-b2a21f70a536 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:23.154824 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 858/1705] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:58:22 2022] DELETE /identity/v3/projects/087b66d74f7a4b67820b8d0872eaa280 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:23.158797 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772e29ab-3d28-41ce-88bc-b2a21f70a536 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:23.160325 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-772e29ab-3d28-41ce-88bc-b2a21f70a536 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:23.224386 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3eab3c-beba-49da-b0c4-d51a9d854e74 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:23.230223 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8037740f-cb99-449a-8849-708feafaaf12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:23.230966 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8037740f-cb99-449a-8849-708feafaaf12 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:23.231240 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8037740f-cb99-449a-8849-708feafaaf12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:23.232759 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8037740f-cb99-449a-8849-708feafaaf12 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:23.232759 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8037740f-cb99-449a-8849-708feafaaf12 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:23.234576 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8037740f-cb99-449a-8849-708feafaaf12 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:23.245657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772e29ab-3d28-41ce-88bc-b2a21f70a536 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:23.256111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a8f752-43ef-4874-ab23-a12e488d05a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:23.257090 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a8f752-43ef-4874-ab23-a12e488d05a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:23.257501 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a8f752-43ef-4874-ab23-a12e488d05a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:23.258309 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a8f752-43ef-4874-ab23-a12e488d05a8 admin admin] PATH_INFO: `/v3/users/88e655dc848e429c9babb23643116bde` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:23.261064 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a8f752-43ef-4874-ab23-a12e488d05a8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=88e655dc848e429c9babb23643116bde)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:23.263570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a8f752-43ef-4874-ab23-a12e488d05a8 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:23.268904 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 848/1706] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:58:23 2022] POST /identity/v3/users => generated 298 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:23.269881 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8037740f-cb99-449a-8849-708feafaaf12 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:23.271572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-8037740f-cb99-449a-8849-708feafaaf12 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:23.285546 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8037740f-cb99-449a-8849-708feafaaf12 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:23.290089 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e787a2-5bb7-498f-8c9d-ec5863e1b219 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:23.290804 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e787a2-5bb7-498f-8c9d-ec5863e1b219 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:23.291213 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e787a2-5bb7-498f-8c9d-ec5863e1b219 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:23.291395 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e787a2-5bb7-498f-8c9d-ec5863e1b219 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:23.292002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e787a2-5bb7-498f-8c9d-ec5863e1b219 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:23.293797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e787a2-5bb7-498f-8c9d-ec5863e1b219 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:23.305244 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 849/1707] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:23 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:23.309377 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e787a2-5bb7-498f-8c9d-ec5863e1b219 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:23.319932 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e787a2-5bb7-498f-8c9d-ec5863e1b219 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:23.323074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-80a8f752-43ef-4874-ab23-a12e488d05a8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '88e655dc848e429c9babb23643116bde'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:23.326016 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac82d95-f351-4129-a95f-b2e54109ad7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:23.326741 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac82d95-f351-4129-a95f-b2e54109ad7d admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:23.327081 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac82d95-f351-4129-a95f-b2e54109ad7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:23.327489 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac82d95-f351-4129-a95f-b2e54109ad7d admin admin] PATH_INFO: `/v3/projects/2c7305376e9941eead2b689e96bf5738/users/49f030c65f2747ccb3a6c20ac6d8e85b/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:23.336663 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-80a8f752-43ef-4874-ab23-a12e488d05a8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '88e655dc848e429c9babb23643116bde'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:23.337566 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-80a8f752-43ef-4874-ab23-a12e488d05a8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '88e655dc848e429c9babb23643116bde'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:23.353313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-80a8f752-43ef-4874-ab23-a12e488d05a8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '88e655dc848e429c9babb23643116bde'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:23.353555 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac82d95-f351-4129-a95f-b2e54109ad7d admin admin] RBAC: Authorizing `identity:create_grant(project_id=2c7305376e9941eead2b689e96bf5738, user_id=49f030c65f2747ccb3a6c20ac6d8e85b, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:23.356488 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac82d95-f351-4129-a95f-b2e54109ad7d admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:23.361388 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-80a8f752-43ef-4874-ab23-a12e488d05a8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '88e655dc848e429c9babb23643116bde'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:23.369436 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 859/1708] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:58:23 2022] DELETE /identity/v3/users/88e655dc848e429c9babb23643116bde => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:23.381026 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 850/1709] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:23 2022] PUT /identity/v3/projects/2c7305376e9941eead2b689e96bf5738/users/49f030c65f2747ccb3a6c20ac6d8e85b/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 00:58:23.385463 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a8f752-43ef-4874-ab23-a12e488d05a8 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:23.386971 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-80a8f752-43ef-4874-ab23-a12e488d05a8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:23.466026 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=85,keystone:INSERT=2,keystone:DELETE=13 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:23.472939 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a8f752-43ef-4874-ab23-a12e488d05a8 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:23.576652 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b540e485-29ee-42c0-bb20-d9a02daffd68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:23.577325 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b540e485-29ee-42c0-bb20-d9a02daffd68 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:23.577635 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b540e485-29ee-42c0-bb20-d9a02daffd68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:23.577939 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b540e485-29ee-42c0-bb20-d9a02daffd68 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:23.578784 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b540e485-29ee-42c0-bb20-d9a02daffd68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:23.581919 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b540e485-29ee-42c0-bb20-d9a02daffd68 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:23.597451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 860/1710] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:23 2022] GET /identity/v3/roles => generated 1420 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:23.601685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac82d95-f351-4129-a95f-b2e54109ad7d admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:23.615109 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac82d95-f351-4129-a95f-b2e54109ad7d admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:23.622303 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1987ad9-9b06-40ad-9bac-01e3d5c2d78f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:23.623131 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1987ad9-9b06-40ad-9bac-01e3d5c2d78f admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:23.623438 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1987ad9-9b06-40ad-9bac-01e3d5c2d78f admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:23.623889 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1987ad9-9b06-40ad-9bac-01e3d5c2d78f admin admin] PATH_INFO: `/v3/projects/2c7305376e9941eead2b689e96bf5738/users/49f030c65f2747ccb3a6c20ac6d8e85b/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:23.627504 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1987ad9-9b06-40ad-9bac-01e3d5c2d78f admin admin] RBAC: Authorizing `identity:create_grant(project_id=2c7305376e9941eead2b689e96bf5738, user_id=49f030c65f2747ccb3a6c20ac6d8e85b, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:23.630536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1987ad9-9b06-40ad-9bac-01e3d5c2d78f admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:23.644365 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 851/1711] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:23 2022] PUT /identity/v3/projects/2c7305376e9941eead2b689e96bf5738/users/49f030c65f2747ccb3a6c20ac6d8e85b/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 00:58:23.656066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b41c59-a8e2-474d-aaf5-8b9b5fb2d855 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:23.656454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b41c59-a8e2-474d-aaf5-8b9b5fb2d855 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:23.657068 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b41c59-a8e2-474d-aaf5-8b9b5fb2d855 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:23.698284 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-e1b41c59-a8e2-474d-aaf5-8b9b5fb2d855 None None] MFA Rules not processed for user `49f030c65f2747ccb3a6c20ac6d8e85b`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:23.699874 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e1b41c59-a8e2-474d-aaf5-8b9b5fb2d855 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:23.813617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 861/1712] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:23 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:23.827369 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1987ad9-9b06-40ad-9bac-01e3d5c2d78f admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:23.830046 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a1987ad9-9b06-40ad-9bac-01e3d5c2d78f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:23.845515 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b41c59-a8e2-474d-aaf5-8b9b5fb2d855 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:23.846954 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e1b41c59-a8e2-474d-aaf5-8b9b5fb2d855 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:23.931207 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1987ad9-9b06-40ad-9bac-01e3d5c2d78f admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:23.936969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bddf46a6-8e8f-4252-988b-19afe51f2abc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:23.937677 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddf46a6-8e8f-4252-988b-19afe51f2abc service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:23.937947 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddf46a6-8e8f-4252-988b-19afe51f2abc service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:23.938214 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddf46a6-8e8f-4252-988b-19afe51f2abc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:23.940133 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-bddf46a6-8e8f-4252-988b-19afe51f2abc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:23.974574 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:23.974945 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b41c59-a8e2-474d-aaf5-8b9b5fb2d855 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:23.982224 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb63e514-5544-43c3-b110-7b68dadc0ce3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:23.983056 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb63e514-5544-43c3-b110-7b68dadc0ce3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:23.983264 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb63e514-5544-43c3-b110-7b68dadc0ce3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:23.983887 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb63e514-5544-43c3-b110-7b68dadc0ce3 admin admin] PATH_INFO: `/v3/projects/3dff670d1b1d43b2a3e6a9f41b4e25d5` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:23.986223 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb63e514-5544-43c3-b110-7b68dadc0ce3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3dff670d1b1d43b2a3e6a9f41b4e25d5)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:23.988309 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb63e514-5544-43c3-b110-7b68dadc0ce3 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:24.030830 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:24.043711 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bddf46a6-8e8f-4252-988b-19afe51f2abc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:24.045835 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bddf46a6-8e8f-4252-988b-19afe51f2abc service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:24.058805 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 852/1713] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:23 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:24.073706 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-bb63e514-5544-43c3-b110-7b68dadc0ce3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3dff670d1b1d43b2a3e6a9f41b4e25d5', 'request_id': 'req-bb63e514-5544-43c3-b110-7b68dadc0ce3'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:24.092458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-bb63e514-5544-43c3-b110-7b68dadc0ce3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3dff670d1b1d43b2a3e6a9f41b4e25d5', 'request_id': 'req-bb63e514-5544-43c3-b110-7b68dadc0ce3'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:24.100425 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-bb63e514-5544-43c3-b110-7b68dadc0ce3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3dff670d1b1d43b2a3e6a9f41b4e25d5', 'request_id': 'req-bb63e514-5544-43c3-b110-7b68dadc0ce3'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:24.117808 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-bb63e514-5544-43c3-b110-7b68dadc0ce3 admin admin] The token cache is being invalidate because project 3dff670d1b1d43b2a3e6a9f41b4e25d5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:58:24.118099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-bb63e514-5544-43c3-b110-7b68dadc0ce3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:24.118931 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-bb63e514-5544-43c3-b110-7b68dadc0ce3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:24.120903 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 862/1714] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:58:23 2022] DELETE /identity/v3/projects/3dff670d1b1d43b2a3e6a9f41b4e25d5 => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:24.126558 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffbcd82-72d9-420e-bec4-652a7821b896 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:24.127389 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffbcd82-72d9-420e-bec4-652a7821b896 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:24.127665 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffbcd82-72d9-420e-bec4-652a7821b896 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:24.166373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-1ffbcd82-72d9-420e-bec4-652a7821b896 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:24.169638 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-1ffbcd82-72d9-420e-bec4-652a7821b896 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:24.263769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 853/1715] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:24.271066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536f70b4-d196-4edd-8f68-619d3ac00eb9 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:24.271590 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536f70b4-d196-4edd-8f68-619d3ac00eb9 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:24.271590 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536f70b4-d196-4edd-8f68-619d3ac00eb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:24.294664 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-536f70b4-d196-4edd-8f68-619d3ac00eb9 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:24.295906 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-536f70b4-d196-4edd-8f68-619d3ac00eb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:24.304448 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 863/1716] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:24.309201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffbcd82-72d9-420e-bec4-652a7821b896 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:24.319577 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffbcd82-72d9-420e-bec4-652a7821b896 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:24.325756 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ff9b62-3b55-4165-b57c-da53a7c50e4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:24.326379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ff9b62-3b55-4165-b57c-da53a7c50e4d admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:24.327307 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ff9b62-3b55-4165-b57c-da53a7c50e4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:24.327657 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ff9b62-3b55-4165-b57c-da53a7c50e4d admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:24.329033 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ff9b62-3b55-4165-b57c-da53a7c50e4d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:58:24.329345 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ff9b62-3b55-4165-b57c-da53a7c50e4d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:24.331962 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ff9b62-3b55-4165-b57c-da53a7c50e4d admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:24.353574 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536f70b4-d196-4edd-8f68-619d3ac00eb9 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:24.355863 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 854/1717] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:58:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:58:24.366940 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536f70b4-d196-4edd-8f68-619d3ac00eb9 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:24.379644 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e91c38e-2905-497c-9739-082e0ee67bcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:24.382043 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e91c38e-2905-497c-9739-082e0ee67bcb admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:24.382303 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e91c38e-2905-497c-9739-082e0ee67bcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:24.382572 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e91c38e-2905-497c-9739-082e0ee67bcb admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:24.383206 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e91c38e-2905-497c-9739-082e0ee67bcb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:24.385467 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e91c38e-2905-497c-9739-082e0ee67bcb admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:24.418043 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 864/1718] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:58:24 2022] POST /identity/v3/projects => generated 373 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:58:24.422335 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ff9b62-3b55-4165-b57c-da53a7c50e4d admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:24.440620 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ff9b62-3b55-4165-b57c-da53a7c50e4d admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:24.501980 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a7ec73-78ab-473e-827a-7cd4b33b59d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:24.502747 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a7ec73-78ab-473e-827a-7cd4b33b59d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:24.502950 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a7ec73-78ab-473e-827a-7cd4b33b59d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:24.503126 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a7ec73-78ab-473e-827a-7cd4b33b59d9 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:24.503797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a7ec73-78ab-473e-827a-7cd4b33b59d9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:24.506611 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a7ec73-78ab-473e-827a-7cd4b33b59d9 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:24.527585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 855/1719] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:58:24 2022] POST /identity/v3/users => generated 287 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:24.532266 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e91c38e-2905-497c-9739-082e0ee67bcb admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:24.542517 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e91c38e-2905-497c-9739-082e0ee67bcb admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:24.547319 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49c9931-7d5f-4e63-8c29-39232421d03b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:24.547949 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49c9931-7d5f-4e63-8c29-39232421d03b admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:24.548224 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49c9931-7d5f-4e63-8c29-39232421d03b admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:24.548489 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49c9931-7d5f-4e63-8c29-39232421d03b admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:24.549123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49c9931-7d5f-4e63-8c29-39232421d03b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:24.550696 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49c9931-7d5f-4e63-8c29-39232421d03b admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:24.560999 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 865/1720] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:24 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:24.565262 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a7ec73-78ab-473e-827a-7cd4b33b59d9 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:24.576029 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a7ec73-78ab-473e-827a-7cd4b33b59d9 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:24.581111 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba640dda-df03-40fc-98b6-ad22fae90197 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:24.581709 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba640dda-df03-40fc-98b6-ad22fae90197 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:24.581994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba640dda-df03-40fc-98b6-ad22fae90197 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:24.582722 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba640dda-df03-40fc-98b6-ad22fae90197 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:24.583586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba640dda-df03-40fc-98b6-ad22fae90197 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:24.585223 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba640dda-df03-40fc-98b6-ad22fae90197 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:24.597462 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 856/1721] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:24 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:24.602101 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49c9931-7d5f-4e63-8c29-39232421d03b admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:24.613548 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49c9931-7d5f-4e63-8c29-39232421d03b admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:24.619432 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bd1817-3a91-44ea-b6c2-17eae6617bb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:24.620228 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bd1817-3a91-44ea-b6c2-17eae6617bb7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:24.620564 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bd1817-3a91-44ea-b6c2-17eae6617bb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:24.620894 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bd1817-3a91-44ea-b6c2-17eae6617bb7 admin admin] PATH_INFO: `/v3/projects/274b2d35f1de475f86e2604ba286fa70/users/d67718d32d704f92a9979454df9ece78/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:24.639537 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bd1817-3a91-44ea-b6c2-17eae6617bb7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=274b2d35f1de475f86e2604ba286fa70, user_id=d67718d32d704f92a9979454df9ece78, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:24.642913 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bd1817-3a91-44ea-b6c2-17eae6617bb7 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:24.664192 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 866/1722] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:24 2022] PUT /identity/v3/projects/274b2d35f1de475f86e2604ba286fa70/users/d67718d32d704f92a9979454df9ece78/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:24.665239 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3a13fc-c57b-49e6-8d43-32551541429d None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:24.665506 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3a13fc-c57b-49e6-8d43-32551541429d None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:24.665801 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3a13fc-c57b-49e6-8d43-32551541429d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:24.700034 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-4a3a13fc-c57b-49e6-8d43-32551541429d None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:24.701293 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-4a3a13fc-c57b-49e6-8d43-32551541429d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:24.721547 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:24.764385 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 857/1723] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:24.777372 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bd1817-3a91-44ea-b6c2-17eae6617bb7 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:24.779224 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-a6bd1817-3a91-44ea-b6c2-17eae6617bb7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:24.863584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bd1817-3a91-44ea-b6c2-17eae6617bb7 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:24.871984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce504826-558d-450a-911c-0c35e7e6c725 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:24.872702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce504826-558d-450a-911c-0c35e7e6c725 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:24.873038 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce504826-558d-450a-911c-0c35e7e6c725 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:24.873366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce504826-558d-450a-911c-0c35e7e6c725 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:24.883092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce504826-558d-450a-911c-0c35e7e6c725 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:24.884702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce504826-558d-450a-911c-0c35e7e6c725 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:24.895038 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 867/1724] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:25.729571 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54ca79e-fea0-4f3b-bf09-0f4d00c7da6d None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:25.730119 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54ca79e-fea0-4f3b-bf09-0f4d00c7da6d None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:25.730522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54ca79e-fea0-4f3b-bf09-0f4d00c7da6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:25.794472 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-d54ca79e-fea0-4f3b-bf09-0f4d00c7da6d None None] MFA Rules not processed for user `d67718d32d704f92a9979454df9ece78`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:25.802052 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d54ca79e-fea0-4f3b-bf09-0f4d00c7da6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:25.973403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 858/1725] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:25 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:25.974550 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce504826-558d-450a-911c-0c35e7e6c725 service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:25.991452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce504826-558d-450a-911c-0c35e7e6c725 service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:25.998816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc33bbe-7e49-4868-a9e4-3a9b0f05940a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:25.999546 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc33bbe-7e49-4868-a9e4-3a9b0f05940a admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:26.000654 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc33bbe-7e49-4868-a9e4-3a9b0f05940a admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:26.002423 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc33bbe-7e49-4868-a9e4-3a9b0f05940a admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:26.002787 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc33bbe-7e49-4868-a9e4-3a9b0f05940a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:26.004923 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc33bbe-7e49-4868-a9e4-3a9b0f05940a admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:26.055003 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 868/1726] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:58:25 2022] POST /identity/v3/projects => generated 373 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:58:26.056540 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54ca79e-fea0-4f3b-bf09-0f4d00c7da6d None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:26.071206 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54ca79e-fea0-4f3b-bf09-0f4d00c7da6d None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:26.177910 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c53bd4-2178-4648-8221-aa5ad27fec24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:26.179644 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c53bd4-2178-4648-8221-aa5ad27fec24 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:26.179780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c53bd4-2178-4648-8221-aa5ad27fec24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:26.180004 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c53bd4-2178-4648-8221-aa5ad27fec24 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:26.181851 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c53bd4-2178-4648-8221-aa5ad27fec24 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:26.184882 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c53bd4-2178-4648-8221-aa5ad27fec24 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:26.223535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 859/1727] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:58:26 2022] POST /identity/v3/users => generated 293 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:26.225069 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc33bbe-7e49-4868-a9e4-3a9b0f05940a admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:26.240973 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc33bbe-7e49-4868-a9e4-3a9b0f05940a admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:26.254685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746c2c3d-af69-402e-843f-e5958d1fdb49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:26.255367 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746c2c3d-af69-402e-843f-e5958d1fdb49 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:26.255642 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746c2c3d-af69-402e-843f-e5958d1fdb49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:26.256531 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746c2c3d-af69-402e-843f-e5958d1fdb49 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:26.257395 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746c2c3d-af69-402e-843f-e5958d1fdb49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:26.259193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746c2c3d-af69-402e-843f-e5958d1fdb49 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:26.294555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 869/1728] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:26 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:26.295916 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c53bd4-2178-4648-8221-aa5ad27fec24 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:26.323184 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c53bd4-2178-4648-8221-aa5ad27fec24 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:26.336994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2c0ed7-2ef1-43fa-a49c-762aab70cf9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:26.338809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2c0ed7-2ef1-43fa-a49c-762aab70cf9e admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:26.339392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2c0ed7-2ef1-43fa-a49c-762aab70cf9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:26.339843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2c0ed7-2ef1-43fa-a49c-762aab70cf9e admin admin] PATH_INFO: `/v3/projects/26308f7dce144bc6bf2ce74bae869ab1/users/f644d11d2be84a71b85828928c0a3c59/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:26.368330 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2c0ed7-2ef1-43fa-a49c-762aab70cf9e admin admin] RBAC: Authorizing `identity:create_grant(project_id=26308f7dce144bc6bf2ce74bae869ab1, user_id=f644d11d2be84a71b85828928c0a3c59, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:26.373068 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2c0ed7-2ef1-43fa-a49c-762aab70cf9e admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:26.406634 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 860/1729] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:26 2022] PUT /identity/v3/projects/26308f7dce144bc6bf2ce74bae869ab1/users/f644d11d2be84a71b85828928c0a3c59/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:26.408491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746c2c3d-af69-402e-843f-e5958d1fdb49 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:26.424627 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746c2c3d-af69-402e-843f-e5958d1fdb49 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:26.468525 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:26.528892 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf7f120-c574-4f7d-ad2f-e5c91a46176a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:26.529534 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf7f120-c574-4f7d-ad2f-e5c91a46176a admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:26.529827 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf7f120-c574-4f7d-ad2f-e5c91a46176a admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:26.530099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf7f120-c574-4f7d-ad2f-e5c91a46176a admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:26.531302 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf7f120-c574-4f7d-ad2f-e5c91a46176a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:26.532889 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf7f120-c574-4f7d-ad2f-e5c91a46176a admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:26.548324 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 870/1730] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:26 2022] GET /identity/v3/roles => generated 1420 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:26.552678 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2c0ed7-2ef1-43fa-a49c-762aab70cf9e admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:26.567363 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2c0ed7-2ef1-43fa-a49c-762aab70cf9e admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:26.574955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b6233f-a08b-4866-b5cc-155557c48033 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:26.575789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b6233f-a08b-4866-b5cc-155557c48033 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:26.576031 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b6233f-a08b-4866-b5cc-155557c48033 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:26.576440 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b6233f-a08b-4866-b5cc-155557c48033 admin admin] PATH_INFO: `/v3/projects/26308f7dce144bc6bf2ce74bae869ab1/users/f644d11d2be84a71b85828928c0a3c59/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:26.578541 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b6233f-a08b-4866-b5cc-155557c48033 admin admin] RBAC: Authorizing `identity:create_grant(project_id=26308f7dce144bc6bf2ce74bae869ab1, user_id=f644d11d2be84a71b85828928c0a3c59, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:26.581532 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b6233f-a08b-4866-b5cc-155557c48033 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:26.628002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 861/1731] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:26 2022] PUT /identity/v3/projects/26308f7dce144bc6bf2ce74bae869ab1/users/f644d11d2be84a71b85828928c0a3c59/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:26.868546 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf7f120-c574-4f7d-ad2f-e5c91a46176a admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:26.873121 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-abf7f120-c574-4f7d-ad2f-e5c91a46176a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:26.957572 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf7f120-c574-4f7d-ad2f-e5c91a46176a admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:27.015803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e45fff6-7741-4766-9dda-4eed57b73564 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:27.016740 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e45fff6-7741-4766-9dda-4eed57b73564 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:27.017248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e45fff6-7741-4766-9dda-4eed57b73564 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:27.017749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e45fff6-7741-4766-9dda-4eed57b73564 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:27.021715 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-9e45fff6-7741-4766-9dda-4eed57b73564 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:27.231391 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e45fff6-7741-4766-9dda-4eed57b73564 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:27.232910 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e45fff6-7741-4766-9dda-4eed57b73564 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:27.252139 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 871/1732] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:26 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:27.807451 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaa111d-694d-4644-a19e-430906f0c756 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:27.807451 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaa111d-694d-4644-a19e-430906f0c756 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:27.807451 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaa111d-694d-4644-a19e-430906f0c756 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:27.837432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:27.849090 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-afaa111d-694d-4644-a19e-430906f0c756 None None] MFA Rules not processed for user `f644d11d2be84a71b85828928c0a3c59`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:27.851098 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-afaa111d-694d-4644-a19e-430906f0c756 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:28.006486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 862/1733] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:27 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:28.020961 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e45fff6-7741-4766-9dda-4eed57b73564 service cinder] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:28.038707 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e45fff6-7741-4766-9dda-4eed57b73564 service cinder] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:28.045435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273230e3-5366-4a60-b2dd-030d0345a292 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:28.046122 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273230e3-5366-4a60-b2dd-030d0345a292 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:28.046552 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273230e3-5366-4a60-b2dd-030d0345a292 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:28.046891 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273230e3-5366-4a60-b2dd-030d0345a292 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:28.118007 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:28.201992 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273230e3-5366-4a60-b2dd-030d0345a292 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:28.203962 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273230e3-5366-4a60-b2dd-030d0345a292 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:28.222286 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 872/1734] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:28 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:28.357291 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afaa111d-694d-4644-a19e-430906f0c756 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:28.358876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-afaa111d-694d-4644-a19e-430906f0c756 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:28.385268 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afaa111d-694d-4644-a19e-430906f0c756 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:28.396818 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ceb31b-a781-479d-9b9e-4a87c6796263 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:28.397764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ceb31b-a781-479d-9b9e-4a87c6796263 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:28.398486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ceb31b-a781-479d-9b9e-4a87c6796263 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:28.398940 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ceb31b-a781-479d-9b9e-4a87c6796263 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:28.404249 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-46ceb31b-a781-479d-9b9e-4a87c6796263 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:28.521828 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ceb31b-a781-479d-9b9e-4a87c6796263 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:28.523375 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ceb31b-a781-479d-9b9e-4a87c6796263 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:28.551378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 863/1735] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:28 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:29.970658 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273230e3-5366-4a60-b2dd-030d0345a292 service cinder] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:29.980359 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273230e3-5366-4a60-b2dd-030d0345a292 service cinder] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:29.986293 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f7f823-d24f-492f-9364-6ad00fa6db82 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:29.987556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f7f823-d24f-492f-9364-6ad00fa6db82 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:29.988036 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f7f823-d24f-492f-9364-6ad00fa6db82 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:29.988468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f7f823-d24f-492f-9364-6ad00fa6db82 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:30.002161 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f7f823-d24f-492f-9364-6ad00fa6db82 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:30.003945 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f7f823-d24f-492f-9364-6ad00fa6db82 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:30.018935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 873/1736] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:29 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:37.093464 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ceb31b-a781-479d-9b9e-4a87c6796263 service cinder] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:37.096551 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-46ceb31b-a781-479d-9b9e-4a87c6796263 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:37.173888 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ceb31b-a781-479d-9b9e-4a87c6796263 service cinder] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:37.178139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a53906-dfd4-42e7-a502-e1cebb01b349 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:37.178771 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a53906-dfd4-42e7-a502-e1cebb01b349 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:37.179208 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a53906-dfd4-42e7-a502-e1cebb01b349 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:37.179481 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a53906-dfd4-42e7-a502-e1cebb01b349 admin admin] PATH_INFO: `/v3/users/9df7cddf69454122a5a3f5d3dd2162cd` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:37.182029 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a53906-dfd4-42e7-a502-e1cebb01b349 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9df7cddf69454122a5a3f5d3dd2162cd)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:37.184301 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a53906-dfd4-42e7-a502-e1cebb01b349 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:37.229539 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:37.251820 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b6a53906-dfd4-42e7-a502-e1cebb01b349 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9df7cddf69454122a5a3f5d3dd2162cd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:37.264992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b6a53906-dfd4-42e7-a502-e1cebb01b349 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9df7cddf69454122a5a3f5d3dd2162cd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:37.265687 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b6a53906-dfd4-42e7-a502-e1cebb01b349 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9df7cddf69454122a5a3f5d3dd2162cd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:37.282038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b6a53906-dfd4-42e7-a502-e1cebb01b349 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9df7cddf69454122a5a3f5d3dd2162cd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:37.300407 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b6a53906-dfd4-42e7-a502-e1cebb01b349 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9df7cddf69454122a5a3f5d3dd2162cd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:37.313925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 864/1737] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:58:37 2022] DELETE /identity/v3/users/9df7cddf69454122a5a3f5d3dd2162cd => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:37.315793 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1174a4c-adf8-43fd-9667-4579cdd0be84 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:37.316593 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1174a4c-adf8-43fd-9667-4579cdd0be84 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:37.317070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1174a4c-adf8-43fd-9667-4579cdd0be84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:37.349830 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-e1174a4c-adf8-43fd-9667-4579cdd0be84 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:37.351343 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e1174a4c-adf8-43fd-9667-4579cdd0be84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:37.436151 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 874/1738] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:37.505931 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a53906-dfd4-42e7-a502-e1cebb01b349 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:37.507684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b6a53906-dfd4-42e7-a502-e1cebb01b349 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:37.575859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a53906-dfd4-42e7-a502-e1cebb01b349 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:37.579595 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26835f56-7dda-4857-a74b-3b6c64789242 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:37.580333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26835f56-7dda-4857-a74b-3b6c64789242 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:37.580666 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26835f56-7dda-4857-a74b-3b6c64789242 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:37.580991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26835f56-7dda-4857-a74b-3b6c64789242 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:37.593329 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26835f56-7dda-4857-a74b-3b6c64789242 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:37.595483 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26835f56-7dda-4857-a74b-3b6c64789242 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:37.614101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 865/1739] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:38.839137 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1174a4c-adf8-43fd-9667-4579cdd0be84 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:38.846288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e1174a4c-adf8-43fd-9667-4579cdd0be84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:38.862163 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1174a4c-adf8-43fd-9667-4579cdd0be84 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:38.868405 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f87b680-1839-4902-ac9c-b82ac8637b2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:38.869838 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f87b680-1839-4902-ac9c-b82ac8637b2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:38.870175 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f87b680-1839-4902-ac9c-b82ac8637b2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:38.870651 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f87b680-1839-4902-ac9c-b82ac8637b2c admin admin] PATH_INFO: `/v3/projects/2c8e4152267645dc9b8537ec5109d314` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:38.871943 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f87b680-1839-4902-ac9c-b82ac8637b2c admin admin] RBAC: Authorizing `identity:delete_project(project_id=2c8e4152267645dc9b8537ec5109d314)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:38.875387 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f87b680-1839-4902-ac9c-b82ac8637b2c admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:38.930162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8f87b680-1839-4902-ac9c-b82ac8637b2c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2c8e4152267645dc9b8537ec5109d314', 'request_id': 'req-8f87b680-1839-4902-ac9c-b82ac8637b2c'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:38.939057 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8f87b680-1839-4902-ac9c-b82ac8637b2c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2c8e4152267645dc9b8537ec5109d314', 'request_id': 'req-8f87b680-1839-4902-ac9c-b82ac8637b2c'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:38.944600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8f87b680-1839-4902-ac9c-b82ac8637b2c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2c8e4152267645dc9b8537ec5109d314', 'request_id': 'req-8f87b680-1839-4902-ac9c-b82ac8637b2c'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:38.956992 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8f87b680-1839-4902-ac9c-b82ac8637b2c admin admin] The token cache is being invalidate because project 2c8e4152267645dc9b8537ec5109d314 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:58:38.957276 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8f87b680-1839-4902-ac9c-b82ac8637b2c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:38.957933 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8f87b680-1839-4902-ac9c-b82ac8637b2c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:38.959610 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 875/1740] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:58:38 2022] DELETE /identity/v3/projects/2c8e4152267645dc9b8537ec5109d314 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:38.963769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26835f56-7dda-4857-a74b-3b6c64789242 service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:38.973665 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26835f56-7dda-4857-a74b-3b6c64789242 service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:39.029732 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937786aa-c664-418a-aefd-27ce833a56f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:39.030257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937786aa-c664-418a-aefd-27ce833a56f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:39.030529 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937786aa-c664-418a-aefd-27ce833a56f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:39.030795 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937786aa-c664-418a-aefd-27ce833a56f1 admin admin] PATH_INFO: `/v3/users/fb0ba64e4211465ea31c12abe6470168` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:39.032000 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937786aa-c664-418a-aefd-27ce833a56f1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fb0ba64e4211465ea31c12abe6470168)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:39.033920 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937786aa-c664-418a-aefd-27ce833a56f1 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:39.084267 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-937786aa-c664-418a-aefd-27ce833a56f1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fb0ba64e4211465ea31c12abe6470168'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:39.087918 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:39.093528 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-937786aa-c664-418a-aefd-27ce833a56f1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fb0ba64e4211465ea31c12abe6470168'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:39.094952 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-937786aa-c664-418a-aefd-27ce833a56f1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fb0ba64e4211465ea31c12abe6470168'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:39.103769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-937786aa-c664-418a-aefd-27ce833a56f1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fb0ba64e4211465ea31c12abe6470168'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:39.115837 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-937786aa-c664-418a-aefd-27ce833a56f1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fb0ba64e4211465ea31c12abe6470168'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:39.122194 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 866/1741] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:58:38 2022] DELETE /identity/v3/users/fb0ba64e4211465ea31c12abe6470168 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:39.402797 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f87b680-1839-4902-ac9c-b82ac8637b2c admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:39.404270 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-8f87b680-1839-4902-ac9c-b82ac8637b2c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:39.426359 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f87b680-1839-4902-ac9c-b82ac8637b2c admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:39.464869 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:39.495502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77448e11-86e2-436e-aa92-83c1393e533c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:39.496173 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77448e11-86e2-436e-aa92-83c1393e533c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:39.496700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77448e11-86e2-436e-aa92-83c1393e533c admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:39.496973 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77448e11-86e2-436e-aa92-83c1393e533c admin admin] PATH_INFO: `/v3/projects/0c52a0b9b1704717b7d6c6ab88947b6b` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:39.498302 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77448e11-86e2-436e-aa92-83c1393e533c admin admin] RBAC: Authorizing `identity:delete_project(project_id=0c52a0b9b1704717b7d6c6ab88947b6b)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:39.501156 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77448e11-86e2-436e-aa92-83c1393e533c admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:39.553644 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-77448e11-86e2-436e-aa92-83c1393e533c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0c52a0b9b1704717b7d6c6ab88947b6b', 'request_id': 'req-77448e11-86e2-436e-aa92-83c1393e533c'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:39.561896 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-77448e11-86e2-436e-aa92-83c1393e533c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0c52a0b9b1704717b7d6c6ab88947b6b', 'request_id': 'req-77448e11-86e2-436e-aa92-83c1393e533c'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:39.569154 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-77448e11-86e2-436e-aa92-83c1393e533c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0c52a0b9b1704717b7d6c6ab88947b6b', 'request_id': 'req-77448e11-86e2-436e-aa92-83c1393e533c'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:39.581119 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-77448e11-86e2-436e-aa92-83c1393e533c admin admin] The token cache is being invalidate because project 0c52a0b9b1704717b7d6c6ab88947b6b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:58:39.581313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-77448e11-86e2-436e-aa92-83c1393e533c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:39.581991 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-77448e11-86e2-436e-aa92-83c1393e533c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:39.583512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 876/1742] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:58:39 2022] DELETE /identity/v3/projects/0c52a0b9b1704717b7d6c6ab88947b6b => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:39.587577 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937786aa-c664-418a-aefd-27ce833a56f1 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:39.589173 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-937786aa-c664-418a-aefd-27ce833a56f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:39.602815 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937786aa-c664-418a-aefd-27ce833a56f1 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:39.660918 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b245987-bf83-4e01-8a93-ea6dc4cbf17d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:39.661695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b245987-bf83-4e01-8a93-ea6dc4cbf17d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:39.661993 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b245987-bf83-4e01-8a93-ea6dc4cbf17d admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:39.662378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b245987-bf83-4e01-8a93-ea6dc4cbf17d admin admin] PATH_INFO: `/v3/users/49f030c65f2747ccb3a6c20ac6d8e85b` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:39.663476 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b245987-bf83-4e01-8a93-ea6dc4cbf17d admin admin] RBAC: Authorizing `identity:delete_user(user_id=49f030c65f2747ccb3a6c20ac6d8e85b)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:39.665423 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b245987-bf83-4e01-8a93-ea6dc4cbf17d admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:39.716698 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-2b245987-bf83-4e01-8a93-ea6dc4cbf17d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '49f030c65f2747ccb3a6c20ac6d8e85b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:39.726244 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-2b245987-bf83-4e01-8a93-ea6dc4cbf17d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '49f030c65f2747ccb3a6c20ac6d8e85b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:39.726885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-2b245987-bf83-4e01-8a93-ea6dc4cbf17d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '49f030c65f2747ccb3a6c20ac6d8e85b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:39.733999 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-2b245987-bf83-4e01-8a93-ea6dc4cbf17d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '49f030c65f2747ccb3a6c20ac6d8e85b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:39.748639 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-2b245987-bf83-4e01-8a93-ea6dc4cbf17d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '49f030c65f2747ccb3a6c20ac6d8e85b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:39.755130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 867/1743] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:58:39 2022] DELETE /identity/v3/users/49f030c65f2747ccb3a6c20ac6d8e85b => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:40.237972 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77448e11-86e2-436e-aa92-83c1393e533c admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:40.239454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-77448e11-86e2-436e-aa92-83c1393e533c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:40.254834 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77448e11-86e2-436e-aa92-83c1393e533c admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:40.313727 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884bc48d-e3d2-407b-9eeb-a52086618b33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:40.314364 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884bc48d-e3d2-407b-9eeb-a52086618b33 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:40.314671 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884bc48d-e3d2-407b-9eeb-a52086618b33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:40.314942 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884bc48d-e3d2-407b-9eeb-a52086618b33 admin admin] PATH_INFO: `/v3/projects/2c7305376e9941eead2b689e96bf5738` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:40.316176 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884bc48d-e3d2-407b-9eeb-a52086618b33 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2c7305376e9941eead2b689e96bf5738)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:40.318157 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884bc48d-e3d2-407b-9eeb-a52086618b33 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:40.369246 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-884bc48d-e3d2-407b-9eeb-a52086618b33 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2c7305376e9941eead2b689e96bf5738', 'request_id': 'req-884bc48d-e3d2-407b-9eeb-a52086618b33'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:40.378394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-884bc48d-e3d2-407b-9eeb-a52086618b33 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2c7305376e9941eead2b689e96bf5738', 'request_id': 'req-884bc48d-e3d2-407b-9eeb-a52086618b33'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:40.384988 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-884bc48d-e3d2-407b-9eeb-a52086618b33 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2c7305376e9941eead2b689e96bf5738', 'request_id': 'req-884bc48d-e3d2-407b-9eeb-a52086618b33'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:40.386969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:40.398269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-884bc48d-e3d2-407b-9eeb-a52086618b33 admin admin] The token cache is being invalidate because project 2c7305376e9941eead2b689e96bf5738 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:58:40.398554 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-884bc48d-e3d2-407b-9eeb-a52086618b33 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:40.399212 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-884bc48d-e3d2-407b-9eeb-a52086618b33 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:40.400738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 877/1744] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:58:40 2022] DELETE /identity/v3/projects/2c7305376e9941eead2b689e96bf5738 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:40.405750 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b416a866-dc45-4ad9-b490-f5880981c21a None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:40.406291 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b416a866-dc45-4ad9-b490-f5880981c21a None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:40.406291 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b416a866-dc45-4ad9-b490-f5880981c21a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:40.425716 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-b416a866-dc45-4ad9-b490-f5880981c21a None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:40.426910 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b416a866-dc45-4ad9-b490-f5880981c21a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:40.473773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:40.497412 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 868/1745] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:40.500058 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6c2600-09e6-4126-a301-aa4149e045a7 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:40.500445 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6c2600-09e6-4126-a301-aa4149e045a7 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:40.500791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6c2600-09e6-4126-a301-aa4149e045a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:40.520738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-dd6c2600-09e6-4126-a301-aa4149e045a7 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:40.521933 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-dd6c2600-09e6-4126-a301-aa4149e045a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:40.530876 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 878/1746] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:58:40.535471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b416a866-dc45-4ad9-b490-f5880981c21a None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:40.545240 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b416a866-dc45-4ad9-b490-f5880981c21a None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:40.550016 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e84d4f9-7e5f-4cdb-8bc4-1692d3b3edc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:40.550616 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e84d4f9-7e5f-4cdb-8bc4-1692d3b3edc5 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:40.550888 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e84d4f9-7e5f-4cdb-8bc4-1692d3b3edc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:40.551154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e84d4f9-7e5f-4cdb-8bc4-1692d3b3edc5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:40.551826 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e84d4f9-7e5f-4cdb-8bc4-1692d3b3edc5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:58:40.552125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e84d4f9-7e5f-4cdb-8bc4-1692d3b3edc5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:40.553710 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e84d4f9-7e5f-4cdb-8bc4-1692d3b3edc5 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:40.566116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 869/1747] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:58:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:58:40.570908 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6c2600-09e6-4126-a301-aa4149e045a7 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:40.580664 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6c2600-09e6-4126-a301-aa4149e045a7 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:40.585773 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ebc612-b90d-4913-80d1-958a4c03c13d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:40.586370 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ebc612-b90d-4913-80d1-958a4c03c13d admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:40.586639 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ebc612-b90d-4913-80d1-958a4c03c13d admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:40.586898 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ebc612-b90d-4913-80d1-958a4c03c13d admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:40.587511 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ebc612-b90d-4913-80d1-958a4c03c13d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:40.589443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ebc612-b90d-4913-80d1-958a4c03c13d admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:40.623171 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 879/1748] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:58:40 2022] POST /identity/v3/projects => generated 373 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:40.627517 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e84d4f9-7e5f-4cdb-8bc4-1692d3b3edc5 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:40.638538 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e84d4f9-7e5f-4cdb-8bc4-1692d3b3edc5 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:40.701719 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d55e7f-aee5-404d-bbc9-8282b8963cac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:40.702320 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d55e7f-aee5-404d-bbc9-8282b8963cac admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:40.702589 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d55e7f-aee5-404d-bbc9-8282b8963cac admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:40.702848 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d55e7f-aee5-404d-bbc9-8282b8963cac admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:40.703512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d55e7f-aee5-404d-bbc9-8282b8963cac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:40.706805 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d55e7f-aee5-404d-bbc9-8282b8963cac admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:40.720110 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ebc612-b90d-4913-80d1-958a4c03c13d admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:40.721947 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-14ebc612-b90d-4913-80d1-958a4c03c13d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:40.730373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 870/1749] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:58:40 2022] POST /identity/v3/users => generated 287 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:40.734179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d55e7f-aee5-404d-bbc9-8282b8963cac admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:40.735822 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ebc612-b90d-4913-80d1-958a4c03c13d admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:40.740376 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a001d91a-c7e6-43d6-a9c7-64302ecfc45f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:40.741002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a001d91a-c7e6-43d6-a9c7-64302ecfc45f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:40.741277 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a001d91a-c7e6-43d6-a9c7-64302ecfc45f admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:40.741541 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a001d91a-c7e6-43d6-a9c7-64302ecfc45f admin admin] PATH_INFO: `/v3/users/0dfb161b6a9144f9aa0d4cf47d90ab20` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:40.742648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a001d91a-c7e6-43d6-a9c7-64302ecfc45f admin admin] RBAC: Authorizing `identity:delete_user(user_id=0dfb161b6a9144f9aa0d4cf47d90ab20)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:40.743878 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d55e7f-aee5-404d-bbc9-8282b8963cac admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:40.744592 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a001d91a-c7e6-43d6-a9c7-64302ecfc45f admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:40.748457 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa833dc-d327-41a1-8e06-5ffbe90185ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:40.749038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa833dc-d327-41a1-8e06-5ffbe90185ef admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:40.749317 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa833dc-d327-41a1-8e06-5ffbe90185ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:40.749583 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa833dc-d327-41a1-8e06-5ffbe90185ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:40.750237 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa833dc-d327-41a1-8e06-5ffbe90185ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:40.751835 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa833dc-d327-41a1-8e06-5ffbe90185ef admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:40.763366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 871/1750] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:40 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:40.767886 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa833dc-d327-41a1-8e06-5ffbe90185ef admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:40.779316 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa833dc-d327-41a1-8e06-5ffbe90185ef admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:40.784582 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155be492-0f0a-4710-96df-c2e6d4f5b28e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:40.785202 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155be492-0f0a-4710-96df-c2e6d4f5b28e admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:40.785477 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155be492-0f0a-4710-96df-c2e6d4f5b28e admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:40.785721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155be492-0f0a-4710-96df-c2e6d4f5b28e admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:40.786350 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155be492-0f0a-4710-96df-c2e6d4f5b28e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:40.788075 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155be492-0f0a-4710-96df-c2e6d4f5b28e admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:40.792761 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a001d91a-c7e6-43d6-a9c7-64302ecfc45f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0dfb161b6a9144f9aa0d4cf47d90ab20'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:40.798701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 872/1751] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:40 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 00:58:40.801313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a001d91a-c7e6-43d6-a9c7-64302ecfc45f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0dfb161b6a9144f9aa0d4cf47d90ab20'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:40.802501 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a001d91a-c7e6-43d6-a9c7-64302ecfc45f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0dfb161b6a9144f9aa0d4cf47d90ab20'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:40.802870 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155be492-0f0a-4710-96df-c2e6d4f5b28e admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:40.804577 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-155be492-0f0a-4710-96df-c2e6d4f5b28e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:40.814793 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a001d91a-c7e6-43d6-a9c7-64302ecfc45f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0dfb161b6a9144f9aa0d4cf47d90ab20'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:40.817579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155be492-0f0a-4710-96df-c2e6d4f5b28e admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:40.822146 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c566b138-912d-42c3-a77e-359dedece006 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:40.822780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c566b138-912d-42c3-a77e-359dedece006 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:40.823366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c566b138-912d-42c3-a77e-359dedece006 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:40.823629 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c566b138-912d-42c3-a77e-359dedece006 admin admin] PATH_INFO: `/v3/projects/dd16277cbc344c9aa8ffbc45ce1a16d2/users/5f198d5e999b487bad75a6b423fe1adc/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:40.825433 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a001d91a-c7e6-43d6-a9c7-64302ecfc45f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0dfb161b6a9144f9aa0d4cf47d90ab20'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:40.831732 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 880/1752] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:58:40 2022] DELETE /identity/v3/users/0dfb161b6a9144f9aa0d4cf47d90ab20 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 00:58:40.836103 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fba185-788e-41b3-b40c-f5040407e128 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:40.836413 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fba185-788e-41b3-b40c-f5040407e128 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:40.836685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fba185-788e-41b3-b40c-f5040407e128 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:40.843274 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c566b138-912d-42c3-a77e-359dedece006 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dd16277cbc344c9aa8ffbc45ce1a16d2, user_id=5f198d5e999b487bad75a6b423fe1adc, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:40.846202 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c566b138-912d-42c3-a77e-359dedece006 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:40.855442 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-c7fba185-788e-41b3-b40c-f5040407e128 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:40.856661 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c7fba185-788e-41b3-b40c-f5040407e128 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:40.860893 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 873/1753] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:40 2022] PUT /identity/v3/projects/dd16277cbc344c9aa8ffbc45ce1a16d2/users/5f198d5e999b487bad75a6b423fe1adc/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 00:58:40.866636 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac53945f-0b16-4b20-b087-cda3125a44de None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:40.866915 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac53945f-0b16-4b20-b087-cda3125a44de None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:40.867208 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac53945f-0b16-4b20-b087-cda3125a44de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:40.887534 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-ac53945f-0b16-4b20-b087-cda3125a44de None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:40.888651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-ac53945f-0b16-4b20-b087-cda3125a44de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:40.914767 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 881/1754] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:58:40.925369 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fba185-788e-41b3-b40c-f5040407e128 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:40.926794 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c7fba185-788e-41b3-b40c-f5040407e128 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:40.950493 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 874/1755] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:40.960882 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac53945f-0b16-4b20-b087-cda3125a44de None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:40.962510 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-ac53945f-0b16-4b20-b087-cda3125a44de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:40.971995 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:40.987237 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:40.987593 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fba185-788e-41b3-b40c-f5040407e128 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:40.991617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a180347-891c-4edb-8981-0dea398fda8e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:40.992277 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a180347-891c-4edb-8981-0dea398fda8e service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:40.992556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a180347-891c-4edb-8981-0dea398fda8e service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:40.992823 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a180347-891c-4edb-8981-0dea398fda8e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:41.002842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a180347-891c-4edb-8981-0dea398fda8e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:41.004313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a180347-891c-4edb-8981-0dea398fda8e service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:41.013887 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 882/1756] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 00:58:41.026648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac53945f-0b16-4b20-b087-cda3125a44de None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:41.030450 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8de12f-2675-4be2-93a3-79a5994c9318 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:41.031058 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8de12f-2675-4be2-93a3-79a5994c9318 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:41.031435 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8de12f-2675-4be2-93a3-79a5994c9318 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:41.031935 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8de12f-2675-4be2-93a3-79a5994c9318 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:41.041434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8de12f-2675-4be2-93a3-79a5994c9318 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:41.042866 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c8de12f-2675-4be2-93a3-79a5994c9318 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:41.052983 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 875/1757] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:41.514399 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a180347-891c-4edb-8981-0dea398fda8e service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:41.515928 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-0a180347-891c-4edb-8981-0dea398fda8e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:41.538004 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a180347-891c-4edb-8981-0dea398fda8e service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:41.543486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2eaf27-e143-4c43-a8e5-23c5f06c3b08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:41.544288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2eaf27-e143-4c43-a8e5-23c5f06c3b08 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:41.544692 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2eaf27-e143-4c43-a8e5-23c5f06c3b08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:41.545027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2eaf27-e143-4c43-a8e5-23c5f06c3b08 admin admin] PATH_INFO: `/v3/projects/4370999c56184197856c3b8b25fd2e20` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:41.547260 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2eaf27-e143-4c43-a8e5-23c5f06c3b08 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4370999c56184197856c3b8b25fd2e20)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:41.549961 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2eaf27-e143-4c43-a8e5-23c5f06c3b08 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:41.682709 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ff2eaf27-e143-4c43-a8e5-23c5f06c3b08 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4370999c56184197856c3b8b25fd2e20', 'request_id': 'req-ff2eaf27-e143-4c43-a8e5-23c5f06c3b08'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:41.709416 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ff2eaf27-e143-4c43-a8e5-23c5f06c3b08 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4370999c56184197856c3b8b25fd2e20', 'request_id': 'req-ff2eaf27-e143-4c43-a8e5-23c5f06c3b08'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:41.722207 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ff2eaf27-e143-4c43-a8e5-23c5f06c3b08 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4370999c56184197856c3b8b25fd2e20', 'request_id': 'req-ff2eaf27-e143-4c43-a8e5-23c5f06c3b08'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:41.737389 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ff2eaf27-e143-4c43-a8e5-23c5f06c3b08 admin admin] The token cache is being invalidate because project 4370999c56184197856c3b8b25fd2e20 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:58:41.737923 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ff2eaf27-e143-4c43-a8e5-23c5f06c3b08 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:41.738959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ff2eaf27-e143-4c43-a8e5-23c5f06c3b08 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:41.741117 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 883/1758] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:58:41 2022] DELETE /identity/v3/projects/4370999c56184197856c3b8b25fd2e20 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:41.745844 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6418205-b1e7-4df1-b0e4-e9ee05e9fa97 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:41.746314 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6418205-b1e7-4df1-b0e4-e9ee05e9fa97 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:41.746700 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6418205-b1e7-4df1-b0e4-e9ee05e9fa97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:41.770889 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-b6418205-b1e7-4df1-b0e4-e9ee05e9fa97 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:41.772194 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b6418205-b1e7-4df1-b0e4-e9ee05e9fa97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:41.847434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 876/1759] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:41.855261 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d20909a-1ad9-4149-b6a7-b18df71c535e None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:41.855538 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d20909a-1ad9-4149-b6a7-b18df71c535e None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:41.855839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d20909a-1ad9-4149-b6a7-b18df71c535e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:41.881158 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-2d20909a-1ad9-4149-b6a7-b18df71c535e None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:41.883286 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-2d20909a-1ad9-4149-b6a7-b18df71c535e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:41.898576 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 884/1760] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:41.902724 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6418205-b1e7-4df1-b0e4-e9ee05e9fa97 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:41.913831 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6418205-b1e7-4df1-b0e4-e9ee05e9fa97 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:41.919317 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c553e9e-fe9b-475c-893a-1d016633dac3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:41.919748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c553e9e-fe9b-475c-893a-1d016633dac3 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:41.920038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c553e9e-fe9b-475c-893a-1d016633dac3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:41.920299 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c553e9e-fe9b-475c-893a-1d016633dac3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:41.920944 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c553e9e-fe9b-475c-893a-1d016633dac3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:58:41.921237 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c553e9e-fe9b-475c-893a-1d016633dac3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:41.923019 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c553e9e-fe9b-475c-893a-1d016633dac3 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:41.936245 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 877/1761] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:58:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 00:58:41.941571 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d20909a-1ad9-4149-b6a7-b18df71c535e None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:41.956391 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d20909a-1ad9-4149-b6a7-b18df71c535e None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:41.965540 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8b80fa-b2a5-4902-a605-6e8b88ba7f4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:41.966598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8b80fa-b2a5-4902-a605-6e8b88ba7f4d admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:41.966959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8b80fa-b2a5-4902-a605-6e8b88ba7f4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:41.967365 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8b80fa-b2a5-4902-a605-6e8b88ba7f4d admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:41.968021 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8b80fa-b2a5-4902-a605-6e8b88ba7f4d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:41.970330 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8b80fa-b2a5-4902-a605-6e8b88ba7f4d admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:42.032549 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f6ffb7-f38a-499e-8237-ddc564c06d27 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:42.032903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f6ffb7-f38a-499e-8237-ddc564c06d27 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:42.033135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f6ffb7-f38a-499e-8237-ddc564c06d27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:42.034241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 885/1762] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:58:41 2022] POST /identity/v3/projects => generated 373 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:42.038073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8b80fa-b2a5-4902-a605-6e8b88ba7f4d admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:42.077482 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-46f6ffb7-f38a-499e-8237-ddc564c06d27 None None] MFA Rules not processed for user `5f198d5e999b487bad75a6b423fe1adc`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:42.078726 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-46f6ffb7-f38a-499e-8237-ddc564c06d27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:42.130260 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8b80fa-b2a5-4902-a605-6e8b88ba7f4d admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:42.135598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734c9955-5345-4c0a-8db3-b0f333329e24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:42.136517 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734c9955-5345-4c0a-8db3-b0f333329e24 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:42.137003 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734c9955-5345-4c0a-8db3-b0f333329e24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:42.137477 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734c9955-5345-4c0a-8db3-b0f333329e24 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:42.138319 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734c9955-5345-4c0a-8db3-b0f333329e24 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:42.140690 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734c9955-5345-4c0a-8db3-b0f333329e24 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:42.171854 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 886/1763] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:58:42 2022] POST /identity/v3/users => generated 287 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:58:42.176751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734c9955-5345-4c0a-8db3-b0f333329e24 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:42.183839 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 878/1764] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:42 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:42.190429 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734c9955-5345-4c0a-8db3-b0f333329e24 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:42.194699 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f6ffb7-f38a-499e-8237-ddc564c06d27 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:42.195164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b573a636-d443-4ea6-aaf0-dc5141d767ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:42.195812 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b573a636-d443-4ea6-aaf0-dc5141d767ae admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:42.196091 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b573a636-d443-4ea6-aaf0-dc5141d767ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:42.196205 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-46f6ffb7-f38a-499e-8237-ddc564c06d27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:42.196408 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b573a636-d443-4ea6-aaf0-dc5141d767ae admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:42.196996 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b573a636-d443-4ea6-aaf0-dc5141d767ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:42.198893 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b573a636-d443-4ea6-aaf0-dc5141d767ae admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:42.210064 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 887/1765] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:42 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:42.213994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b573a636-d443-4ea6-aaf0-dc5141d767ae admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:42.223845 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b573a636-d443-4ea6-aaf0-dc5141d767ae admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:42.230276 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b782a6-6c89-4daa-9cae-46295ac33cd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:42.230875 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b782a6-6c89-4daa-9cae-46295ac33cd2 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:42.231142 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b782a6-6c89-4daa-9cae-46295ac33cd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:42.231404 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b782a6-6c89-4daa-9cae-46295ac33cd2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:42.232069 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b782a6-6c89-4daa-9cae-46295ac33cd2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:42.233777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b782a6-6c89-4daa-9cae-46295ac33cd2 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:42.246137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:42.246923 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 888/1766] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:42 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 00:58:42.252784 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b782a6-6c89-4daa-9cae-46295ac33cd2 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:42.259040 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:42.262400 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f6ffb7-f38a-499e-8237-ddc564c06d27 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:42.263564 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b782a6-6c89-4daa-9cae-46295ac33cd2 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:42.266203 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0fb077-2e26-432a-b619-eae7a39e527d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:42.266904 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0fb077-2e26-432a-b619-eae7a39e527d service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:42.267196 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0fb077-2e26-432a-b619-eae7a39e527d service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:42.267578 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0fb077-2e26-432a-b619-eae7a39e527d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:42.271775 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1547ce77-4736-4e86-a386-2d20795e8eb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:42.273710 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1547ce77-4736-4e86-a386-2d20795e8eb8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:42.274040 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1547ce77-4736-4e86-a386-2d20795e8eb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:42.274556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1547ce77-4736-4e86-a386-2d20795e8eb8 admin admin] PATH_INFO: `/v3/projects/8b1114cdfa2e4d5aaece48c405eba0ee/users/75f3c899744d415eab82df852568139e/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:42.278045 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d0fb077-2e26-432a-b619-eae7a39e527d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:42.280095 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d0fb077-2e26-432a-b619-eae7a39e527d service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:42.290338 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 879/1767] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:42 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 00:58:42.292740 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1547ce77-4736-4e86-a386-2d20795e8eb8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8b1114cdfa2e4d5aaece48c405eba0ee, user_id=75f3c899744d415eab82df852568139e, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:42.295564 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1547ce77-4736-4e86-a386-2d20795e8eb8 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:42.312870 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 889/1768] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:42 2022] PUT /identity/v3/projects/8b1114cdfa2e4d5aaece48c405eba0ee/users/75f3c899744d415eab82df852568139e/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:42.324370 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051f7fa9-761e-4721-b393-4dc6b53697d3 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:42.324664 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051f7fa9-761e-4721-b393-4dc6b53697d3 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:42.325000 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051f7fa9-761e-4721-b393-4dc6b53697d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:42.362218 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-051f7fa9-761e-4721-b393-4dc6b53697d3 None None] MFA Rules not processed for user `75f3c899744d415eab82df852568139e`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:42.363550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-051f7fa9-761e-4721-b393-4dc6b53697d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:42.441833 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 880/1769] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:42 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:42.452875 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1547ce77-4736-4e86-a386-2d20795e8eb8 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:42.454656 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-1547ce77-4736-4e86-a386-2d20795e8eb8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:42.519073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1547ce77-4736-4e86-a386-2d20795e8eb8 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:42.523092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33ba9a3-57fd-4522-8474-dd058c3d7dab service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:42.523797 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33ba9a3-57fd-4522-8474-dd058c3d7dab service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:42.524077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33ba9a3-57fd-4522-8474-dd058c3d7dab service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:42.524370 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33ba9a3-57fd-4522-8474-dd058c3d7dab service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:42.536256 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33ba9a3-57fd-4522-8474-dd058c3d7dab service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:42.537835 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33ba9a3-57fd-4522-8474-dd058c3d7dab service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:42.549198 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 890/1770] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:42 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:43.734468 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560eefe9-dd88-4238-9d98-1a9db3173737 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:43.735007 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560eefe9-dd88-4238-9d98-1a9db3173737 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:43.735007 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560eefe9-dd88-4238-9d98-1a9db3173737 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:43.755180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-560eefe9-dd88-4238-9d98-1a9db3173737 None None] MFA Rules not processed for user `5f198d5e999b487bad75a6b423fe1adc`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:43.756407 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-560eefe9-dd88-4238-9d98-1a9db3173737 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:43.823566 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 881/1771] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:43 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:46.832312 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33ba9a3-57fd-4522-8474-dd058c3d7dab service cinder] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:46.903274 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33ba9a3-57fd-4522-8474-dd058c3d7dab service cinder] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:46.908091 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e983ccc8-6f57-4241-a604-62fddaf52518 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:46.908718 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e983ccc8-6f57-4241-a604-62fddaf52518 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:46.908994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e983ccc8-6f57-4241-a604-62fddaf52518 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:46.909256 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e983ccc8-6f57-4241-a604-62fddaf52518 admin admin] PATH_INFO: `/v3/users/75f3c899744d415eab82df852568139e` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:46.910356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e983ccc8-6f57-4241-a604-62fddaf52518 admin admin] RBAC: Authorizing `identity:delete_user(user_id=75f3c899744d415eab82df852568139e)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:46.912298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e983ccc8-6f57-4241-a604-62fddaf52518 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:46.965234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-e983ccc8-6f57-4241-a604-62fddaf52518 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '75f3c899744d415eab82df852568139e'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:46.974578 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-e983ccc8-6f57-4241-a604-62fddaf52518 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '75f3c899744d415eab82df852568139e'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:46.975359 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-e983ccc8-6f57-4241-a604-62fddaf52518 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '75f3c899744d415eab82df852568139e'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:46.987634 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-e983ccc8-6f57-4241-a604-62fddaf52518 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '75f3c899744d415eab82df852568139e'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:46.994436 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-e983ccc8-6f57-4241-a604-62fddaf52518 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '75f3c899744d415eab82df852568139e'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:47.000672 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 891/1772] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:58:46 2022] DELETE /identity/v3/users/75f3c899744d415eab82df852568139e => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:47.006328 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2961920d-7bfb-4a06-af30-3170aebd0664 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:47.006796 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2961920d-7bfb-4a06-af30-3170aebd0664 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:47.006987 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2961920d-7bfb-4a06-af30-3170aebd0664 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:47.028867 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-2961920d-7bfb-4a06-af30-3170aebd0664 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:47.030256 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-2961920d-7bfb-4a06-af30-3170aebd0664 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:47.102818 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 882/1773] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:58:47.112763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e983ccc8-6f57-4241-a604-62fddaf52518 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:47.114339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e983ccc8-6f57-4241-a604-62fddaf52518 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:47.123969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:47.185857 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e983ccc8-6f57-4241-a604-62fddaf52518 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:47.189498 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88d1651-53fd-4b92-9732-c5c5e7891294 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:47.190109 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88d1651-53fd-4b92-9732-c5c5e7891294 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:47.190393 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88d1651-53fd-4b92-9732-c5c5e7891294 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:47.190659 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88d1651-53fd-4b92-9732-c5c5e7891294 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:47.200160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88d1651-53fd-4b92-9732-c5c5e7891294 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:47.201591 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88d1651-53fd-4b92-9732-c5c5e7891294 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:47.213437 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 892/1774] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:47.570987 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2961920d-7bfb-4a06-af30-3170aebd0664 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:47.575776 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-2961920d-7bfb-4a06-af30-3170aebd0664 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:47.615147 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2961920d-7bfb-4a06-af30-3170aebd0664 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:47.627519 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fb4ff8-7e3b-439d-a71c-5c9a925e12a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:47.628466 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fb4ff8-7e3b-439d-a71c-5c9a925e12a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:47.628721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fb4ff8-7e3b-439d-a71c-5c9a925e12a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:47.629010 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fb4ff8-7e3b-439d-a71c-5c9a925e12a9 admin admin] PATH_INFO: `/v3/projects/8b1114cdfa2e4d5aaece48c405eba0ee` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:47.630460 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fb4ff8-7e3b-439d-a71c-5c9a925e12a9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8b1114cdfa2e4d5aaece48c405eba0ee)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:47.632535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fb4ff8-7e3b-439d-a71c-5c9a925e12a9 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:47.649270 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:47.712402 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-33fb4ff8-7e3b-439d-a71c-5c9a925e12a9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8b1114cdfa2e4d5aaece48c405eba0ee', 'request_id': 'req-33fb4ff8-7e3b-439d-a71c-5c9a925e12a9'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:47.720283 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-33fb4ff8-7e3b-439d-a71c-5c9a925e12a9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8b1114cdfa2e4d5aaece48c405eba0ee', 'request_id': 'req-33fb4ff8-7e3b-439d-a71c-5c9a925e12a9'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:47.730983 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-33fb4ff8-7e3b-439d-a71c-5c9a925e12a9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8b1114cdfa2e4d5aaece48c405eba0ee', 'request_id': 'req-33fb4ff8-7e3b-439d-a71c-5c9a925e12a9'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:47.745481 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-33fb4ff8-7e3b-439d-a71c-5c9a925e12a9 admin admin] The token cache is being invalidate because project 8b1114cdfa2e4d5aaece48c405eba0ee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:58:47.746835 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-33fb4ff8-7e3b-439d-a71c-5c9a925e12a9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:47.748022 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-33fb4ff8-7e3b-439d-a71c-5c9a925e12a9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:47.749921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 883/1775] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:58:47 2022] DELETE /identity/v3/projects/8b1114cdfa2e4d5aaece48c405eba0ee => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:47.755373 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5faf992f-de89-4eeb-a8b4-18dafd3dde4e None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:47.755664 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5faf992f-de89-4eeb-a8b4-18dafd3dde4e None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:47.755983 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5faf992f-de89-4eeb-a8b4-18dafd3dde4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:47.778010 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-5faf992f-de89-4eeb-a8b4-18dafd3dde4e None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:47.779590 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-5faf992f-de89-4eeb-a8b4-18dafd3dde4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:47.875918 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 893/1776] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:47.876614 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea70578-1077-4c59-b01d-d911dd1a0bab None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:47.876614 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea70578-1077-4c59-b01d-d911dd1a0bab None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:47.876614 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea70578-1077-4c59-b01d-d911dd1a0bab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:47.899753 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-2ea70578-1077-4c59-b01d-d911dd1a0bab None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:47.900990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-2ea70578-1077-4c59-b01d-d911dd1a0bab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:47.910438 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 884/1777] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:47.914871 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5faf992f-de89-4eeb-a8b4-18dafd3dde4e None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:47.928358 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5faf992f-de89-4eeb-a8b4-18dafd3dde4e None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:47.933481 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98977f26-512e-49c1-9faf-4bfabf99d1ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:47.933994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98977f26-512e-49c1-9faf-4bfabf99d1ad admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:47.934268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98977f26-512e-49c1-9faf-4bfabf99d1ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:47.934533 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98977f26-512e-49c1-9faf-4bfabf99d1ad admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:47.935174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98977f26-512e-49c1-9faf-4bfabf99d1ad admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:58:47.935479 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98977f26-512e-49c1-9faf-4bfabf99d1ad admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:47.940225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98977f26-512e-49c1-9faf-4bfabf99d1ad admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:47.954895 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 894/1778] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:58:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:58:47.960536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea70578-1077-4c59-b01d-d911dd1a0bab None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:47.972137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea70578-1077-4c59-b01d-d911dd1a0bab None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:47.984574 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e3f240-3f54-46f8-af26-e91191a702c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:47.985244 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e3f240-3f54-46f8-af26-e91191a702c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:47.985516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e3f240-3f54-46f8-af26-e91191a702c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:47.985786 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e3f240-3f54-46f8-af26-e91191a702c6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:47.986436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e3f240-3f54-46f8-af26-e91191a702c6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:47.988401 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e3f240-3f54-46f8-af26-e91191a702c6 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:48.029860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 885/1779] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:58:47 2022] POST /identity/v3/projects => generated 389 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:48.032185 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98977f26-512e-49c1-9faf-4bfabf99d1ad admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:48.045948 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98977f26-512e-49c1-9faf-4bfabf99d1ad admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:48.149141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f87bdd-f90a-4fda-98ca-ea02a8caefa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:48.149936 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f87bdd-f90a-4fda-98ca-ea02a8caefa0 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:48.150114 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f87bdd-f90a-4fda-98ca-ea02a8caefa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:48.150363 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f87bdd-f90a-4fda-98ca-ea02a8caefa0 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:48.152072 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f87bdd-f90a-4fda-98ca-ea02a8caefa0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:48.154598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f87bdd-f90a-4fda-98ca-ea02a8caefa0 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:48.186542 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 895/1780] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:58:48 2022] POST /identity/v3/users => generated 295 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:48.188409 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e3f240-3f54-46f8-af26-e91191a702c6 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:48.201183 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e3f240-3f54-46f8-af26-e91191a702c6 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:48.208475 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e453748-65cb-4538-a316-1bd99e3b07c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:48.209148 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e453748-65cb-4538-a316-1bd99e3b07c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:48.209632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e453748-65cb-4538-a316-1bd99e3b07c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:48.210069 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e453748-65cb-4538-a316-1bd99e3b07c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:48.210827 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e453748-65cb-4538-a316-1bd99e3b07c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:48.212765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e453748-65cb-4538-a316-1bd99e3b07c4 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:48.224319 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 886/1781] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:48 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:48.228580 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f87bdd-f90a-4fda-98ca-ea02a8caefa0 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:48.240397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f87bdd-f90a-4fda-98ca-ea02a8caefa0 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:48.244238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718001a1-b9f1-44e5-bb4d-10bcfa195a14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:48.244911 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718001a1-b9f1-44e5-bb4d-10bcfa195a14 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:48.245350 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718001a1-b9f1-44e5-bb4d-10bcfa195a14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:48.245812 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718001a1-b9f1-44e5-bb4d-10bcfa195a14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:48.246485 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718001a1-b9f1-44e5-bb4d-10bcfa195a14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:48.248216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718001a1-b9f1-44e5-bb4d-10bcfa195a14 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:48.261808 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 896/1782] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:48 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:48.265636 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e453748-65cb-4538-a316-1bd99e3b07c4 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:48.280965 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e453748-65cb-4538-a316-1bd99e3b07c4 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:48.295975 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca7c9b3-2c98-498c-8243-49f31d0970ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:48.295975 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca7c9b3-2c98-498c-8243-49f31d0970ba admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:48.295975 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca7c9b3-2c98-498c-8243-49f31d0970ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:48.295975 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca7c9b3-2c98-498c-8243-49f31d0970ba admin admin] PATH_INFO: `/v3/projects/506f3ab4eec248b38c494a68d7f951f1/users/c82d6b7f905a45debdde4323f35ee789/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:48.306355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca7c9b3-2c98-498c-8243-49f31d0970ba admin admin] RBAC: Authorizing `identity:create_grant(project_id=506f3ab4eec248b38c494a68d7f951f1, user_id=c82d6b7f905a45debdde4323f35ee789, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:48.309098 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca7c9b3-2c98-498c-8243-49f31d0970ba admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:48.335989 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 887/1783] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:48 2022] PUT /identity/v3/projects/506f3ab4eec248b38c494a68d7f951f1/users/c82d6b7f905a45debdde4323f35ee789/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:48.348573 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dea329-a6ff-43ae-9ad5-42f66a7faea0 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:48.349738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dea329-a6ff-43ae-9ad5-42f66a7faea0 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:48.350020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dea329-a6ff-43ae-9ad5-42f66a7faea0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:48.377156 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:48.385961 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-50dea329-a6ff-43ae-9ad5-42f66a7faea0 None None] MFA Rules not processed for user `c82d6b7f905a45debdde4323f35ee789`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:48.387998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-50dea329-a6ff-43ae-9ad5-42f66a7faea0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:48.480479 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 897/1784] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:48 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:48.496544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca7c9b3-2c98-498c-8243-49f31d0970ba admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:48.498324 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-0ca7c9b3-2c98-498c-8243-49f31d0970ba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:48.570370 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca7c9b3-2c98-498c-8243-49f31d0970ba admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:48.574738 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c07979-a241-44ab-9baf-2e8ffe6d6dcb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:48.575672 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c07979-a241-44ab-9baf-2e8ffe6d6dcb service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:48.576044 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c07979-a241-44ab-9baf-2e8ffe6d6dcb service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:48.576391 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c07979-a241-44ab-9baf-2e8ffe6d6dcb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:48.585675 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c07979-a241-44ab-9baf-2e8ffe6d6dcb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:48.587132 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c07979-a241-44ab-9baf-2e8ffe6d6dcb service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:48.603959 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 888/1785] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:48 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:50.818797 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50dea329-a6ff-43ae-9ad5-42f66a7faea0 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:50.820406 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-50dea329-a6ff-43ae-9ad5-42f66a7faea0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:50.889390 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50dea329-a6ff-43ae-9ad5-42f66a7faea0 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:50.895066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a8dcd1-c49c-42b3-8add-8b8908477e44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:50.895779 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a8dcd1-c49c-42b3-8add-8b8908477e44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:50.896044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a8dcd1-c49c-42b3-8add-8b8908477e44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:50.896310 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a8dcd1-c49c-42b3-8add-8b8908477e44 admin admin] PATH_INFO: `/v3/users/d67718d32d704f92a9979454df9ece78` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:50.897942 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a8dcd1-c49c-42b3-8add-8b8908477e44 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d67718d32d704f92a9979454df9ece78)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:50.900139 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a8dcd1-c49c-42b3-8add-8b8908477e44 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:50.949028 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-26a8dcd1-c49c-42b3-8add-8b8908477e44 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd67718d32d704f92a9979454df9ece78'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:50.958090 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-26a8dcd1-c49c-42b3-8add-8b8908477e44 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd67718d32d704f92a9979454df9ece78'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:50.958735 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-26a8dcd1-c49c-42b3-8add-8b8908477e44 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd67718d32d704f92a9979454df9ece78'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:50.970859 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-26a8dcd1-c49c-42b3-8add-8b8908477e44 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd67718d32d704f92a9979454df9ece78'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:50.979976 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-26a8dcd1-c49c-42b3-8add-8b8908477e44 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd67718d32d704f92a9979454df9ece78'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:50.986414 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 898/1786] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:58:50 2022] DELETE /identity/v3/users/d67718d32d704f92a9979454df9ece78 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:51.169246 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c07979-a241-44ab-9baf-2e8ffe6d6dcb service cinder] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:51.180572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c07979-a241-44ab-9baf-2e8ffe6d6dcb service cinder] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:51.350521 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:51.371194 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ac3fb4-67e0-4107-8b38-6440e99b144b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:51.371892 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ac3fb4-67e0-4107-8b38-6440e99b144b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:51.372162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ac3fb4-67e0-4107-8b38-6440e99b144b admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:51.372425 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ac3fb4-67e0-4107-8b38-6440e99b144b admin admin] PATH_INFO: `/v3/projects/274b2d35f1de475f86e2604ba286fa70` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:51.373548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ac3fb4-67e0-4107-8b38-6440e99b144b admin admin] RBAC: Authorizing `identity:delete_project(project_id=274b2d35f1de475f86e2604ba286fa70)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:51.375638 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ac3fb4-67e0-4107-8b38-6440e99b144b admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:51.427282 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-72ac3fb4-67e0-4107-8b38-6440e99b144b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '274b2d35f1de475f86e2604ba286fa70', 'request_id': 'req-72ac3fb4-67e0-4107-8b38-6440e99b144b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:51.433844 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-72ac3fb4-67e0-4107-8b38-6440e99b144b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '274b2d35f1de475f86e2604ba286fa70', 'request_id': 'req-72ac3fb4-67e0-4107-8b38-6440e99b144b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:51.443898 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-72ac3fb4-67e0-4107-8b38-6440e99b144b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '274b2d35f1de475f86e2604ba286fa70', 'request_id': 'req-72ac3fb4-67e0-4107-8b38-6440e99b144b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:51.461489 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-72ac3fb4-67e0-4107-8b38-6440e99b144b admin admin] The token cache is being invalidate because project 274b2d35f1de475f86e2604ba286fa70 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:58:51.461778 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-72ac3fb4-67e0-4107-8b38-6440e99b144b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:51.462511 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-72ac3fb4-67e0-4107-8b38-6440e99b144b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:51.464184 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 889/1787] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:58:51 2022] DELETE /identity/v3/projects/274b2d35f1de475f86e2604ba286fa70 => generated 0 bytes in 296 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:51.468811 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a8dcd1-c49c-42b3-8add-8b8908477e44 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:51.470459 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-26a8dcd1-c49c-42b3-8add-8b8908477e44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:51.483739 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a8dcd1-c49c-42b3-8add-8b8908477e44 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:51.504813 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:51.552732 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4920e2b8-8432-417d-afeb-0a0f90736753 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:51.553403 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4920e2b8-8432-417d-afeb-0a0f90736753 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:51.553685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4920e2b8-8432-417d-afeb-0a0f90736753 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:51.553989 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4920e2b8-8432-417d-afeb-0a0f90736753 admin admin] PATH_INFO: `/v3/users/f644d11d2be84a71b85828928c0a3c59` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:51.555647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4920e2b8-8432-417d-afeb-0a0f90736753 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f644d11d2be84a71b85828928c0a3c59)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:51.557201 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4920e2b8-8432-417d-afeb-0a0f90736753 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:51.613591 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-4920e2b8-8432-417d-afeb-0a0f90736753 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f644d11d2be84a71b85828928c0a3c59'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:51.626961 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-4920e2b8-8432-417d-afeb-0a0f90736753 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f644d11d2be84a71b85828928c0a3c59'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:51.627642 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-4920e2b8-8432-417d-afeb-0a0f90736753 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f644d11d2be84a71b85828928c0a3c59'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:51.648000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-4920e2b8-8432-417d-afeb-0a0f90736753 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f644d11d2be84a71b85828928c0a3c59'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:51.656030 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-4920e2b8-8432-417d-afeb-0a0f90736753 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f644d11d2be84a71b85828928c0a3c59'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:51.666609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 899/1788] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:58:51 2022] DELETE /identity/v3/users/f644d11d2be84a71b85828928c0a3c59 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:51.845942 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ac3fb4-67e0-4107-8b38-6440e99b144b admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:51.847441 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-72ac3fb4-67e0-4107-8b38-6440e99b144b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:51.862514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ac3fb4-67e0-4107-8b38-6440e99b144b admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:51.920308 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcfd1d8-6e3a-46eb-aaac-b04576756307 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:51.920931 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcfd1d8-6e3a-46eb-aaac-b04576756307 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:51.921204 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcfd1d8-6e3a-46eb-aaac-b04576756307 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:51.921468 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcfd1d8-6e3a-46eb-aaac-b04576756307 admin admin] PATH_INFO: `/v3/projects/26308f7dce144bc6bf2ce74bae869ab1` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:51.922774 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbcfd1d8-6e3a-46eb-aaac-b04576756307 admin admin] RBAC: Authorizing `identity:delete_project(project_id=26308f7dce144bc6bf2ce74bae869ab1)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:51.924780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbcfd1d8-6e3a-46eb-aaac-b04576756307 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:51.970428 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-fbcfd1d8-6e3a-46eb-aaac-b04576756307 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '26308f7dce144bc6bf2ce74bae869ab1', 'request_id': 'req-fbcfd1d8-6e3a-46eb-aaac-b04576756307'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:51.976024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-fbcfd1d8-6e3a-46eb-aaac-b04576756307 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '26308f7dce144bc6bf2ce74bae869ab1', 'request_id': 'req-fbcfd1d8-6e3a-46eb-aaac-b04576756307'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:51.986166 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-fbcfd1d8-6e3a-46eb-aaac-b04576756307 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '26308f7dce144bc6bf2ce74bae869ab1', 'request_id': 'req-fbcfd1d8-6e3a-46eb-aaac-b04576756307'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:51.994679 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:51.998720 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-fbcfd1d8-6e3a-46eb-aaac-b04576756307 admin admin] The token cache is being invalidate because project 26308f7dce144bc6bf2ce74bae869ab1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:58:51.999116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-fbcfd1d8-6e3a-46eb-aaac-b04576756307 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:52.000118 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-fbcfd1d8-6e3a-46eb-aaac-b04576756307 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:52.001835 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 890/1789] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:58:51 2022] DELETE /identity/v3/projects/26308f7dce144bc6bf2ce74bae869ab1 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:52.006926 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51181be-66ec-4f7c-88af-04e9a94d2925 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:52.007435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51181be-66ec-4f7c-88af-04e9a94d2925 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:52.007633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51181be-66ec-4f7c-88af-04e9a94d2925 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:52.028660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-a51181be-66ec-4f7c-88af-04e9a94d2925 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:52.030298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-a51181be-66ec-4f7c-88af-04e9a94d2925 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:52.095131 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 900/1790] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:52.107371 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593db6db-a530-40c6-9a08-fc765ffb3cde None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:52.107550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593db6db-a530-40c6-9a08-fc765ffb3cde None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:52.107884 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593db6db-a530-40c6-9a08-fc765ffb3cde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:52.133524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-593db6db-a530-40c6-9a08-fc765ffb3cde None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:52.134765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-593db6db-a530-40c6-9a08-fc765ffb3cde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:52.145040 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 891/1791] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:52.149572 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51181be-66ec-4f7c-88af-04e9a94d2925 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:52.162752 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51181be-66ec-4f7c-88af-04e9a94d2925 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:52.168299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c4d1b4-d40a-4d6e-8a1c-520e68f54af0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:52.168914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c4d1b4-d40a-4d6e-8a1c-520e68f54af0 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:52.169191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c4d1b4-d40a-4d6e-8a1c-520e68f54af0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:52.169441 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c4d1b4-d40a-4d6e-8a1c-520e68f54af0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:52.170092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c4d1b4-d40a-4d6e-8a1c-520e68f54af0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:58:52.170389 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c4d1b4-d40a-4d6e-8a1c-520e68f54af0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:52.172027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c4d1b4-d40a-4d6e-8a1c-520e68f54af0 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:52.184510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 901/1792] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:58:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:58:52.190126 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593db6db-a530-40c6-9a08-fc765ffb3cde None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:52.205824 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593db6db-a530-40c6-9a08-fc765ffb3cde None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:52.213765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc88e8cb-89a9-4bdb-af4c-58e552706b50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:52.214651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc88e8cb-89a9-4bdb-af4c-58e552706b50 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:52.215087 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc88e8cb-89a9-4bdb-af4c-58e552706b50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:52.215521 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc88e8cb-89a9-4bdb-af4c-58e552706b50 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:52.216491 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc88e8cb-89a9-4bdb-af4c-58e552706b50 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:52.219555 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc88e8cb-89a9-4bdb-af4c-58e552706b50 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:52.249906 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 892/1793] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:58:52 2022] POST /identity/v3/projects => generated 385 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:58:52.254440 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c4d1b4-d40a-4d6e-8a1c-520e68f54af0 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:52.264737 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c4d1b4-d40a-4d6e-8a1c-520e68f54af0 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:52.290823 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:52.330649 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e67197-4cd6-450e-b654-241b4dbf317a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:52.331217 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e67197-4cd6-450e-b654-241b4dbf317a admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:52.331488 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e67197-4cd6-450e-b654-241b4dbf317a admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:52.331782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e67197-4cd6-450e-b654-241b4dbf317a admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:52.332458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e67197-4cd6-450e-b654-241b4dbf317a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:52.337146 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e67197-4cd6-450e-b654-241b4dbf317a admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:52.365234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 902/1794] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:58:52 2022] POST /identity/v3/users => generated 293 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:52.366532 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc88e8cb-89a9-4bdb-af4c-58e552706b50 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:52.377739 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc88e8cb-89a9-4bdb-af4c-58e552706b50 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:52.382674 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1c2b87-85fa-4d99-a611-c55734887ca2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:52.383309 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1c2b87-85fa-4d99-a611-c55734887ca2 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:52.383576 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1c2b87-85fa-4d99-a611-c55734887ca2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:52.383865 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1c2b87-85fa-4d99-a611-c55734887ca2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:52.384556 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1c2b87-85fa-4d99-a611-c55734887ca2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:52.386270 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1c2b87-85fa-4d99-a611-c55734887ca2 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:52.397752 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 893/1795] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:52 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:52.403628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e67197-4cd6-450e-b654-241b4dbf317a admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:52.417910 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e67197-4cd6-450e-b654-241b4dbf317a admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:52.423978 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e6e761-2876-4132-b708-35c8210448db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:52.424594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e6e761-2876-4132-b708-35c8210448db admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:52.424875 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e6e761-2876-4132-b708-35c8210448db admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:52.425168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e6e761-2876-4132-b708-35c8210448db admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:52.425848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e6e761-2876-4132-b708-35c8210448db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:52.427453 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e6e761-2876-4132-b708-35c8210448db admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:52.438227 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 903/1796] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:52 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 00:58:52.444087 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1c2b87-85fa-4d99-a611-c55734887ca2 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:52.455833 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1c2b87-85fa-4d99-a611-c55734887ca2 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:52.461416 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d557be0-2cf2-4dfd-b448-cab3ecacdd37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:52.462095 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d557be0-2cf2-4dfd-b448-cab3ecacdd37 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:52.462371 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d557be0-2cf2-4dfd-b448-cab3ecacdd37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:52.462639 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d557be0-2cf2-4dfd-b448-cab3ecacdd37 admin admin] PATH_INFO: `/v3/projects/e9c6ee5d75d845d1b7d4cf0b7d1b8e25/users/ee1812dd3a274e83a7d2a72fd2cc5662/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:52.479138 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d557be0-2cf2-4dfd-b448-cab3ecacdd37 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e9c6ee5d75d845d1b7d4cf0b7d1b8e25, user_id=ee1812dd3a274e83a7d2a72fd2cc5662, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:52.481879 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d557be0-2cf2-4dfd-b448-cab3ecacdd37 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:52.497284 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 894/1797] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:52 2022] PUT /identity/v3/projects/e9c6ee5d75d845d1b7d4cf0b7d1b8e25/users/ee1812dd3a274e83a7d2a72fd2cc5662/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:52.509429 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203977d0-2225-479c-9cf5-c10ce4e905b8 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:52.509714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203977d0-2225-479c-9cf5-c10ce4e905b8 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:52.509988 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203977d0-2225-479c-9cf5-c10ce4e905b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:52.538180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-203977d0-2225-479c-9cf5-c10ce4e905b8 None None] MFA Rules not processed for user `ee1812dd3a274e83a7d2a72fd2cc5662`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:52.539408 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-203977d0-2225-479c-9cf5-c10ce4e905b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:52.612471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 904/1798] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:52 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:52.624114 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d557be0-2cf2-4dfd-b448-cab3ecacdd37 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:52.626204 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-6d557be0-2cf2-4dfd-b448-cab3ecacdd37 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:52.687788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d557be0-2cf2-4dfd-b448-cab3ecacdd37 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:52.692099 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f90753-e3ad-494a-8b58-6ce2da69be62 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:52.692981 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f90753-e3ad-494a-8b58-6ce2da69be62 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:52.693255 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f90753-e3ad-494a-8b58-6ce2da69be62 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:52.693632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f90753-e3ad-494a-8b58-6ce2da69be62 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:52.703321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f90753-e3ad-494a-8b58-6ce2da69be62 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:52.705086 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f90753-e3ad-494a-8b58-6ce2da69be62 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:52.715461 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 895/1799] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:52 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:52.738197 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203977d0-2225-479c-9cf5-c10ce4e905b8 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:52.816854 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203977d0-2225-479c-9cf5-c10ce4e905b8 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:52.821434 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ead38a-690f-405e-b57c-6f9fd65cb732 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:52.822148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ead38a-690f-405e-b57c-6f9fd65cb732 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:52.822492 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ead38a-690f-405e-b57c-6f9fd65cb732 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:52.822821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ead38a-690f-405e-b57c-6f9fd65cb732 admin admin] PATH_INFO: `/v3/users/ee1812dd3a274e83a7d2a72fd2cc5662` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:52.824380 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ead38a-690f-405e-b57c-6f9fd65cb732 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ee1812dd3a274e83a7d2a72fd2cc5662)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:52.826470 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ead38a-690f-405e-b57c-6f9fd65cb732 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:52.867061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:52.877654 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-55ead38a-690f-405e-b57c-6f9fd65cb732 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ee1812dd3a274e83a7d2a72fd2cc5662'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:52.885601 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-55ead38a-690f-405e-b57c-6f9fd65cb732 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ee1812dd3a274e83a7d2a72fd2cc5662'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:52.886377 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-55ead38a-690f-405e-b57c-6f9fd65cb732 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ee1812dd3a274e83a7d2a72fd2cc5662'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:52.899619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-55ead38a-690f-405e-b57c-6f9fd65cb732 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ee1812dd3a274e83a7d2a72fd2cc5662'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:52.906828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-55ead38a-690f-405e-b57c-6f9fd65cb732 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ee1812dd3a274e83a7d2a72fd2cc5662'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:52.913761 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 905/1800] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:58:52 2022] DELETE /identity/v3/users/ee1812dd3a274e83a7d2a72fd2cc5662 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:52.918278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7983e38-5275-416d-9ac3-6ad30d8c2815 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:52.918628 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7983e38-5275-416d-9ac3-6ad30d8c2815 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:52.918961 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7983e38-5275-416d-9ac3-6ad30d8c2815 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:52.938188 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-f7983e38-5275-416d-9ac3-6ad30d8c2815 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:52.939370 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-f7983e38-5275-416d-9ac3-6ad30d8c2815 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:52.995823 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 896/1801] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:53.006382 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ead38a-690f-405e-b57c-6f9fd65cb732 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:53.007918 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-55ead38a-690f-405e-b57c-6f9fd65cb732 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:53.064942 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ead38a-690f-405e-b57c-6f9fd65cb732 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:53.069188 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48116099-021e-4035-bc12-c7010a9a56db service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:53.069794 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48116099-021e-4035-bc12-c7010a9a56db service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:53.070065 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48116099-021e-4035-bc12-c7010a9a56db service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:53.070324 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48116099-021e-4035-bc12-c7010a9a56db service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:53.080203 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48116099-021e-4035-bc12-c7010a9a56db service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:53.081615 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48116099-021e-4035-bc12-c7010a9a56db service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:53.091215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 906/1802] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:58:53.606009 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7983e38-5275-416d-9ac3-6ad30d8c2815 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:53.607476 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-f7983e38-5275-416d-9ac3-6ad30d8c2815 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:53.620424 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7983e38-5275-416d-9ac3-6ad30d8c2815 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:53.626153 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1116f5-5604-40f0-bec1-26b22f2a8bce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:53.626861 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1116f5-5604-40f0-bec1-26b22f2a8bce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:53.627139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1116f5-5604-40f0-bec1-26b22f2a8bce admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:53.627409 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1116f5-5604-40f0-bec1-26b22f2a8bce admin admin] PATH_INFO: `/v3/projects/e9c6ee5d75d845d1b7d4cf0b7d1b8e25` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:53.629101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1116f5-5604-40f0-bec1-26b22f2a8bce admin admin] RBAC: Authorizing `identity:delete_project(project_id=e9c6ee5d75d845d1b7d4cf0b7d1b8e25)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:53.630829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1116f5-5604-40f0-bec1-26b22f2a8bce admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:53.674162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=96,keystone:INSERT=2,keystone:DELETE=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:53.685428 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-fa1116f5-5604-40f0-bec1-26b22f2a8bce admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e9c6ee5d75d845d1b7d4cf0b7d1b8e25', 'request_id': 'req-fa1116f5-5604-40f0-bec1-26b22f2a8bce'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:53.691519 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-fa1116f5-5604-40f0-bec1-26b22f2a8bce admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e9c6ee5d75d845d1b7d4cf0b7d1b8e25', 'request_id': 'req-fa1116f5-5604-40f0-bec1-26b22f2a8bce'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:53.700405 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-fa1116f5-5604-40f0-bec1-26b22f2a8bce admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e9c6ee5d75d845d1b7d4cf0b7d1b8e25', 'request_id': 'req-fa1116f5-5604-40f0-bec1-26b22f2a8bce'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:53.712357 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-fa1116f5-5604-40f0-bec1-26b22f2a8bce admin admin] The token cache is being invalidate because project e9c6ee5d75d845d1b7d4cf0b7d1b8e25 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:58:53.712635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-fa1116f5-5604-40f0-bec1-26b22f2a8bce admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:53.713256 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-fa1116f5-5604-40f0-bec1-26b22f2a8bce admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:58:53.714754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 897/1803] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:58:53 2022] DELETE /identity/v3/projects/e9c6ee5d75d845d1b7d4cf0b7d1b8e25 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:53.719966 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39797108-9572-46fa-bd66-f7ab1eafcc00 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:53.720251 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39797108-9572-46fa-bd66-f7ab1eafcc00 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:53.720522 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39797108-9572-46fa-bd66-f7ab1eafcc00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:53.740029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-39797108-9572-46fa-bd66-f7ab1eafcc00 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:53.741226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-39797108-9572-46fa-bd66-f7ab1eafcc00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:53.805429 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 907/1804] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:53.814034 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da5fc30-1182-497e-ba4b-b8762e0b3c40 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:53.814344 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da5fc30-1182-497e-ba4b-b8762e0b3c40 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:53.814679 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da5fc30-1182-497e-ba4b-b8762e0b3c40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:53.846053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-5da5fc30-1182-497e-ba4b-b8762e0b3c40 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:53.847344 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-5da5fc30-1182-497e-ba4b-b8762e0b3c40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:53.856874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 898/1805] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:53.861281 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39797108-9572-46fa-bd66-f7ab1eafcc00 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:53.871433 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39797108-9572-46fa-bd66-f7ab1eafcc00 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:53.876968 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0517c94-7aea-4ce6-8ddd-08d830489632 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:53.877597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0517c94-7aea-4ce6-8ddd-08d830489632 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:53.877879 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0517c94-7aea-4ce6-8ddd-08d830489632 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:53.878133 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0517c94-7aea-4ce6-8ddd-08d830489632 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:53.878800 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0517c94-7aea-4ce6-8ddd-08d830489632 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:58:53.879081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0517c94-7aea-4ce6-8ddd-08d830489632 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:53.880953 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0517c94-7aea-4ce6-8ddd-08d830489632 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:53.894685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 908/1806] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:58:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:58:53.898895 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da5fc30-1182-497e-ba4b-b8762e0b3c40 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:53.908465 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da5fc30-1182-497e-ba4b-b8762e0b3c40 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:53.913165 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee315d8-7fd2-4bde-b069-60d481bf105a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:53.913768 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee315d8-7fd2-4bde-b069-60d481bf105a admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:53.914056 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee315d8-7fd2-4bde-b069-60d481bf105a admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:53.914323 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee315d8-7fd2-4bde-b069-60d481bf105a admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:53.914983 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee315d8-7fd2-4bde-b069-60d481bf105a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:53.916924 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee315d8-7fd2-4bde-b069-60d481bf105a admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:53.951960 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 899/1807] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:58:53 2022] POST /identity/v3/projects => generated 385 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:58:53.954801 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0517c94-7aea-4ce6-8ddd-08d830489632 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:53.966285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0517c94-7aea-4ce6-8ddd-08d830489632 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:54.018090 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:58:54.035678 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326bf532-b127-4b08-9b17-e4ec479c6ce5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:54.036336 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326bf532-b127-4b08-9b17-e4ec479c6ce5 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:54.036652 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326bf532-b127-4b08-9b17-e4ec479c6ce5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:54.036925 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326bf532-b127-4b08-9b17-e4ec479c6ce5 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:54.037600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326bf532-b127-4b08-9b17-e4ec479c6ce5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:54.039538 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326bf532-b127-4b08-9b17-e4ec479c6ce5 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:54.068009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 909/1808] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:58:53 2022] POST /identity/v3/users => generated 293 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 00:58:54.070273 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee315d8-7fd2-4bde-b069-60d481bf105a admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:54.082447 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee315d8-7fd2-4bde-b069-60d481bf105a admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:54.087572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c6fca6-58fd-48dd-99e0-ff031b1e684a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:54.088317 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c6fca6-58fd-48dd-99e0-ff031b1e684a admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:54.088664 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c6fca6-58fd-48dd-99e0-ff031b1e684a admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:54.088995 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c6fca6-58fd-48dd-99e0-ff031b1e684a admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:54.089715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c6fca6-58fd-48dd-99e0-ff031b1e684a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:54.091383 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c6fca6-58fd-48dd-99e0-ff031b1e684a admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:54.103685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 900/1809] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:54 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:54.107711 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326bf532-b127-4b08-9b17-e4ec479c6ce5 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:54.118058 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326bf532-b127-4b08-9b17-e4ec479c6ce5 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:54.122678 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b701cf-4ba4-4aa3-8559-e1e833f50cf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:54.123280 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b701cf-4ba4-4aa3-8559-e1e833f50cf4 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:54.123557 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b701cf-4ba4-4aa3-8559-e1e833f50cf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:54.123854 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b701cf-4ba4-4aa3-8559-e1e833f50cf4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:54.124505 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b701cf-4ba4-4aa3-8559-e1e833f50cf4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:54.126468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b701cf-4ba4-4aa3-8559-e1e833f50cf4 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:54.146240 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 910/1810] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:58:54 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:58:54.147742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c6fca6-58fd-48dd-99e0-ff031b1e684a admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:54.164129 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c6fca6-58fd-48dd-99e0-ff031b1e684a admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:54.169786 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9813863-03c0-4c48-9403-4e04acdac737 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:54.170498 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9813863-03c0-4c48-9403-4e04acdac737 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:54.170757 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9813863-03c0-4c48-9403-4e04acdac737 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:54.171029 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9813863-03c0-4c48-9403-4e04acdac737 admin admin] PATH_INFO: `/v3/projects/5d5ab90cd811496e8fc9a1baa05d07f1/users/82ac3c5e694b42acbb736a49662c2173/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:54.191645 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9813863-03c0-4c48-9403-4e04acdac737 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d5ab90cd811496e8fc9a1baa05d07f1, user_id=82ac3c5e694b42acbb736a49662c2173, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:54.194357 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9813863-03c0-4c48-9403-4e04acdac737 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:54.212092 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 901/1811] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:58:54 2022] PUT /identity/v3/projects/5d5ab90cd811496e8fc9a1baa05d07f1/users/82ac3c5e694b42acbb736a49662c2173/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:58:54.226809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfe61c1-cd5e-4938-80d5-ad36cdd9d26f None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:54.227091 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfe61c1-cd5e-4938-80d5-ad36cdd9d26f None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:54.227359 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfe61c1-cd5e-4938-80d5-ad36cdd9d26f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:54.260638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-3cfe61c1-cd5e-4938-80d5-ad36cdd9d26f None None] MFA Rules not processed for user `82ac3c5e694b42acbb736a49662c2173`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:58:54.262062 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3cfe61c1-cd5e-4938-80d5-ad36cdd9d26f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:54.336047 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 911/1812] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:58:54 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:58:54.347673 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9813863-03c0-4c48-9403-4e04acdac737 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:58:54.349491 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d9813863-03c0-4c48-9403-4e04acdac737 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:58:54.412625 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9813863-03c0-4c48-9403-4e04acdac737 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:58:54.416169 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d17d05b-595a-46a3-b401-9cbba023ce20 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:58:54.416773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d17d05b-595a-46a3-b401-9cbba023ce20 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:58:54.417043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d17d05b-595a-46a3-b401-9cbba023ce20 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:58:54.417302 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d17d05b-595a-46a3-b401-9cbba023ce20 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:58:54.425581 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d17d05b-595a-46a3-b401-9cbba023ce20 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:58:54.427003 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d17d05b-595a-46a3-b401-9cbba023ce20 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:58:54.435745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 902/1813] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:58:54 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:59:04.332486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=46,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:59:04.431624 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:DELETE=5,keystone:SELECT=64,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:59:19.550660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cfe61c1-cd5e-4938-80d5-ad36cdd9d26f None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:59:19.622507 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cfe61c1-cd5e-4938-80d5-ad36cdd9d26f None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:59:19.627662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f268bb-65bc-42e6-8b71-4c81c8102822 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:59:19.628391 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f268bb-65bc-42e6-8b71-4c81c8102822 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:59:19.628539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f268bb-65bc-42e6-8b71-4c81c8102822 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:59:19.628810 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f268bb-65bc-42e6-8b71-4c81c8102822 admin admin] PATH_INFO: `/v3/users/82ac3c5e694b42acbb736a49662c2173` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:59:19.630127 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f268bb-65bc-42e6-8b71-4c81c8102822 admin admin] RBAC: Authorizing `identity:delete_user(user_id=82ac3c5e694b42acbb736a49662c2173)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:59:19.632121 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f268bb-65bc-42e6-8b71-4c81c8102822 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:59:19.687237 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-05f268bb-65bc-42e6-8b71-4c81c8102822 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '82ac3c5e694b42acbb736a49662c2173'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:59:19.697569 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-05f268bb-65bc-42e6-8b71-4c81c8102822 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '82ac3c5e694b42acbb736a49662c2173'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:59:19.698267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-05f268bb-65bc-42e6-8b71-4c81c8102822 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '82ac3c5e694b42acbb736a49662c2173'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:59:19.711058 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-05f268bb-65bc-42e6-8b71-4c81c8102822 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '82ac3c5e694b42acbb736a49662c2173'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:59:19.718028 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-05f268bb-65bc-42e6-8b71-4c81c8102822 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '82ac3c5e694b42acbb736a49662c2173'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:59:19.724246 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 912/1814] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:59:19 2022] DELETE /identity/v3/users/82ac3c5e694b42acbb736a49662c2173 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:59:19.729191 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b94b69-1511-4ff5-a713-f5d39e02e489 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:59:19.729683 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b94b69-1511-4ff5-a713-f5d39e02e489 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:59:19.729683 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b94b69-1511-4ff5-a713-f5d39e02e489 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:59:19.749272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-a4b94b69-1511-4ff5-a713-f5d39e02e489 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:59:19.750472 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a4b94b69-1511-4ff5-a713-f5d39e02e489 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:59:19.812981 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 903/1815] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:59:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:59:19.826038 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f268bb-65bc-42e6-8b71-4c81c8102822 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:59:19.827508 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-05f268bb-65bc-42e6-8b71-4c81c8102822 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:59:19.885909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f268bb-65bc-42e6-8b71-4c81c8102822 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:59:19.889830 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e673685-0b66-4658-872e-607ca1003b23 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:59:19.890441 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e673685-0b66-4658-872e-607ca1003b23 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:59:19.890710 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e673685-0b66-4658-872e-607ca1003b23 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:59:19.890970 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e673685-0b66-4658-872e-607ca1003b23 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:59:19.900555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e673685-0b66-4658-872e-607ca1003b23 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:59:19.901984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e673685-0b66-4658-872e-607ca1003b23 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:59:19.912535 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 913/1816] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:59:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:59:20.196317 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b94b69-1511-4ff5-a713-f5d39e02e489 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:59:20.197761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a4b94b69-1511-4ff5-a713-f5d39e02e489 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:59:20.211742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b94b69-1511-4ff5-a713-f5d39e02e489 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:59:20.218446 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ad1e73-7fe3-452f-9e7d-2afca2528529 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:59:20.219648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ad1e73-7fe3-452f-9e7d-2afca2528529 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:59:20.219960 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ad1e73-7fe3-452f-9e7d-2afca2528529 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:59:20.220227 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ad1e73-7fe3-452f-9e7d-2afca2528529 admin admin] PATH_INFO: `/v3/projects/5d5ab90cd811496e8fc9a1baa05d07f1` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:59:20.221366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ad1e73-7fe3-452f-9e7d-2afca2528529 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d5ab90cd811496e8fc9a1baa05d07f1)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:59:20.223290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ad1e73-7fe3-452f-9e7d-2afca2528529 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:59:20.288787 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-57ad1e73-7fe3-452f-9e7d-2afca2528529 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d5ab90cd811496e8fc9a1baa05d07f1', 'request_id': 'req-57ad1e73-7fe3-452f-9e7d-2afca2528529'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:59:20.295025 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-57ad1e73-7fe3-452f-9e7d-2afca2528529 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d5ab90cd811496e8fc9a1baa05d07f1', 'request_id': 'req-57ad1e73-7fe3-452f-9e7d-2afca2528529'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:59:20.304450 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-57ad1e73-7fe3-452f-9e7d-2afca2528529 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d5ab90cd811496e8fc9a1baa05d07f1', 'request_id': 'req-57ad1e73-7fe3-452f-9e7d-2afca2528529'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:59:20.319218 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-57ad1e73-7fe3-452f-9e7d-2afca2528529 admin admin] The token cache is being invalidate because project 5d5ab90cd811496e8fc9a1baa05d07f1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:59:20.319804 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-57ad1e73-7fe3-452f-9e7d-2afca2528529 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:59:20.320534 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-57ad1e73-7fe3-452f-9e7d-2afca2528529 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:59:20.322435 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 904/1817] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:59:20 2022] DELETE /identity/v3/projects/5d5ab90cd811496e8fc9a1baa05d07f1 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:59:20.327556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9792ddf7-9531-406e-a44e-b2480a8a926c None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:59:20.328041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9792ddf7-9531-406e-a44e-b2480a8a926c None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:59:20.328506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9792ddf7-9531-406e-a44e-b2480a8a926c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:59:20.351642 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-9792ddf7-9531-406e-a44e-b2480a8a926c None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:59:20.352866 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-9792ddf7-9531-406e-a44e-b2480a8a926c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:59:20.373931 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:59:20.418912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 914/1818] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:59:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:59:20.426899 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98043621-171c-494b-a460-854a21083834 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:59:20.427178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98043621-171c-494b-a460-854a21083834 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:59:20.427467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98043621-171c-494b-a460-854a21083834 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:59:20.456394 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-98043621-171c-494b-a460-854a21083834 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:59:20.457565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-98043621-171c-494b-a460-854a21083834 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:59:20.470460 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 905/1819] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:59:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:59:20.475042 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9792ddf7-9531-406e-a44e-b2480a8a926c None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:59:20.485094 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9792ddf7-9531-406e-a44e-b2480a8a926c None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:59:20.489673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69442ca-ee5b-42b9-9249-3865f8c187e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:59:20.490281 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69442ca-ee5b-42b9-9249-3865f8c187e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:59:20.490677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69442ca-ee5b-42b9-9249-3865f8c187e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:59:20.490940 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69442ca-ee5b-42b9-9249-3865f8c187e3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:59:20.491577 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69442ca-ee5b-42b9-9249-3865f8c187e3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 00:59:20.491916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69442ca-ee5b-42b9-9249-3865f8c187e3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:59:20.493476 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69442ca-ee5b-42b9-9249-3865f8c187e3 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:59:20.509034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 915/1820] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 00:59:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 00:59:20.513298 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98043621-171c-494b-a460-854a21083834 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:59:20.523777 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98043621-171c-494b-a460-854a21083834 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:59:20.530205 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575980a8-92ce-4417-ba39-64eb83b0a709 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:59:20.533927 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575980a8-92ce-4417-ba39-64eb83b0a709 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:59:20.533927 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575980a8-92ce-4417-ba39-64eb83b0a709 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:59:20.533927 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575980a8-92ce-4417-ba39-64eb83b0a709 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:59:20.533927 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575980a8-92ce-4417-ba39-64eb83b0a709 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:59:20.535032 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575980a8-92ce-4417-ba39-64eb83b0a709 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:59:20.570130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 906/1821] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 00:59:20 2022] POST /identity/v3/projects => generated 373 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:59:20.574537 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69442ca-ee5b-42b9-9249-3865f8c187e3 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:59:20.586062 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69442ca-ee5b-42b9-9249-3865f8c187e3 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:59:20.620803 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575980a8-92ce-4417-ba39-64eb83b0a709 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:59:20.622838 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-575980a8-92ce-4417-ba39-64eb83b0a709 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:59:20.699705 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3dca782-7c5f-48b9-b085-3af2eac0119c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:59:20.700100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dca782-7c5f-48b9-b085-3af2eac0119c admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:59:20.700508 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dca782-7c5f-48b9-b085-3af2eac0119c admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:59:20.700782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dca782-7c5f-48b9-b085-3af2eac0119c admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:59:20.701649 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3dca782-7c5f-48b9-b085-3af2eac0119c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:59:20.704124 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3dca782-7c5f-48b9-b085-3af2eac0119c admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:59:20.717072 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:59:20.732318 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 916/1822] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 00:59:20 2022] POST /identity/v3/users => generated 287 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 00:59:20.736675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3dca782-7c5f-48b9-b085-3af2eac0119c admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:59:20.741246 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575980a8-92ce-4417-ba39-64eb83b0a709 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:59:20.749598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d552097-b304-42f2-8291-e8851af571be service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '63cc035057064805a95e9f12714ebade', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:59:20.750168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3dca782-7c5f-48b9-b085-3af2eac0119c admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:59:20.750388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d552097-b304-42f2-8291-e8851af571be service placement] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:59:20.750661 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d552097-b304-42f2-8291-e8851af571be service placement] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:59:20.751187 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d552097-b304-42f2-8291-e8851af571be service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:59:20.753268 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-9d552097-b304-42f2-8291-e8851af571be service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:59:20.757107 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351198ad-7c55-48ea-8e86-4fb0c028435f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:59:20.757878 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351198ad-7c55-48ea-8e86-4fb0c028435f admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:59:20.758277 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351198ad-7c55-48ea-8e86-4fb0c028435f admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:59:20.758634 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351198ad-7c55-48ea-8e86-4fb0c028435f admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:59:20.759382 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351198ad-7c55-48ea-8e86-4fb0c028435f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:59:20.761298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351198ad-7c55-48ea-8e86-4fb0c028435f admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:59:20.771683 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d552097-b304-42f2-8291-e8851af571be service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:59:20.773501 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d552097-b304-42f2-8291-e8851af571be service placement] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:59:20.775120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 917/1823] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:59:20 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:59:20.780802 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351198ad-7c55-48ea-8e86-4fb0c028435f admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:59:20.785229 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 907/1824] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:59:20 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:59:20.792494 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351198ad-7c55-48ea-8e86-4fb0c028435f admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:59:20.798573 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a343761-3722-4d98-86b1-34d514260af6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:59:20.799194 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a343761-3722-4d98-86b1-34d514260af6 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:59:20.799463 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a343761-3722-4d98-86b1-34d514260af6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:59:20.799737 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a343761-3722-4d98-86b1-34d514260af6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:59:20.800475 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a343761-3722-4d98-86b1-34d514260af6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:59:20.802305 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a343761-3722-4d98-86b1-34d514260af6 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:59:20.815843 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 918/1825] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 00:59:20 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 00:59:20.820033 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d552097-b304-42f2-8291-e8851af571be service placement] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:59:20.831079 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d552097-b304-42f2-8291-e8851af571be service placement] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:59:20.835623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15cfa98-2b48-4c8b-9673-de5233196aa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:59:20.836358 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15cfa98-2b48-4c8b-9673-de5233196aa1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:59:20.836633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15cfa98-2b48-4c8b-9673-de5233196aa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:59:20.836903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15cfa98-2b48-4c8b-9673-de5233196aa1 admin admin] PATH_INFO: `/v3/projects/27fc853acd7b4ef3803aec20a793bd06/users/bc8294d49c654c8f9d8f7539b359df85/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:59:20.853886 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d15cfa98-2b48-4c8b-9673-de5233196aa1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=27fc853acd7b4ef3803aec20a793bd06, user_id=bc8294d49c654c8f9d8f7539b359df85, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:59:20.856834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d15cfa98-2b48-4c8b-9673-de5233196aa1 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:59:20.870985 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 908/1826] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 00:59:20 2022] PUT /identity/v3/projects/27fc853acd7b4ef3803aec20a793bd06/users/bc8294d49c654c8f9d8f7539b359df85/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:59:20.881839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef560858-095b-4ce8-a712-455c49ebd8a7 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:59:20.882120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef560858-095b-4ce8-a712-455c49ebd8a7 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:59:20.882383 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef560858-095b-4ce8-a712-455c49ebd8a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:59:20.913239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-ef560858-095b-4ce8-a712-455c49ebd8a7 None None] MFA Rules not processed for user `bc8294d49c654c8f9d8f7539b359df85`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:59:20.914436 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ef560858-095b-4ce8-a712-455c49ebd8a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:59:20.980530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 919/1827] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:59:20 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:59:20.997002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15cfa98-2b48-4c8b-9673-de5233196aa1 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:59:20.999042 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d15cfa98-2b48-4c8b-9673-de5233196aa1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:59:21.062324 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15cfa98-2b48-4c8b-9673-de5233196aa1 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:59:21.068655 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac037715-1cc5-4575-9655-1fbea72061c5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:59:21.069545 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac037715-1cc5-4575-9655-1fbea72061c5 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:59:21.069861 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac037715-1cc5-4575-9655-1fbea72061c5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:59:21.070209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac037715-1cc5-4575-9655-1fbea72061c5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:59:21.081203 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac037715-1cc5-4575-9655-1fbea72061c5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:59:21.082626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac037715-1cc5-4575-9655-1fbea72061c5 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:59:21.091598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 909/1828] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:59:20 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:59:21.717912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef560858-095b-4ce8-a712-455c49ebd8a7 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:59:21.720118 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ef560858-095b-4ce8-a712-455c49ebd8a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:59:21.733895 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:59:21.801245 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef560858-095b-4ce8-a712-455c49ebd8a7 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:59:21.808022 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67312c3-4f60-49a4-beca-469080e7ce0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:59:21.808657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67312c3-4f60-49a4-beca-469080e7ce0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:59:21.808953 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67312c3-4f60-49a4-beca-469080e7ce0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:59:21.809218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67312c3-4f60-49a4-beca-469080e7ce0b admin admin] PATH_INFO: `/v3/users/c82d6b7f905a45debdde4323f35ee789` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:59:21.810366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67312c3-4f60-49a4-beca-469080e7ce0b admin admin] RBAC: Authorizing `identity:delete_user(user_id=c82d6b7f905a45debdde4323f35ee789)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:59:21.812372 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67312c3-4f60-49a4-beca-469080e7ce0b admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:59:21.863109 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-f67312c3-4f60-49a4-beca-469080e7ce0b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c82d6b7f905a45debdde4323f35ee789'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:59:21.873986 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-f67312c3-4f60-49a4-beca-469080e7ce0b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c82d6b7f905a45debdde4323f35ee789'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:59:21.875199 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-f67312c3-4f60-49a4-beca-469080e7ce0b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c82d6b7f905a45debdde4323f35ee789'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:59:21.892316 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-f67312c3-4f60-49a4-beca-469080e7ce0b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c82d6b7f905a45debdde4323f35ee789'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:59:21.899408 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-f67312c3-4f60-49a4-beca-469080e7ce0b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c82d6b7f905a45debdde4323f35ee789'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:59:21.905285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 920/1829] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 00:59:21 2022] DELETE /identity/v3/users/c82d6b7f905a45debdde4323f35ee789 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:59:21.909909 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4391ca65-a479-4b3a-af41-97df163286fc None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:59:21.910392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4391ca65-a479-4b3a-af41-97df163286fc None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:59:21.910392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4391ca65-a479-4b3a-af41-97df163286fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:59:21.930002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-4391ca65-a479-4b3a-af41-97df163286fc None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:59:21.931182 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-4391ca65-a479-4b3a-af41-97df163286fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:59:21.993241 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 910/1830] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:59:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 00:59:22.018410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67312c3-4f60-49a4-beca-469080e7ce0b admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:59:22.020106 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-f67312c3-4f60-49a4-beca-469080e7ce0b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:59:22.082724 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67312c3-4f60-49a4-beca-469080e7ce0b admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:59:22.086423 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d177204-5ad6-43ab-b45d-624efbb70b5e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:59:22.087053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d177204-5ad6-43ab-b45d-624efbb70b5e service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:59:22.087388 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d177204-5ad6-43ab-b45d-624efbb70b5e service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:59:22.087738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d177204-5ad6-43ab-b45d-624efbb70b5e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:59:22.097276 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d177204-5ad6-43ab-b45d-624efbb70b5e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:59:22.098738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d177204-5ad6-43ab-b45d-624efbb70b5e service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:59:22.108732 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 921/1831] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 00:59:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 00:59:22.393688 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4391ca65-a479-4b3a-af41-97df163286fc None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 00:59:22.395261 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-4391ca65-a479-4b3a-af41-97df163286fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:59:22.408048 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4391ca65-a479-4b3a-af41-97df163286fc None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 00:59:22.413001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6025e502-cd8b-486b-a448-cc08b9084f44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 00:59:22.413713 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6025e502-cd8b-486b-a448-cc08b9084f44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:59:22.414737 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6025e502-cd8b-486b-a448-cc08b9084f44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:59:22.414737 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6025e502-cd8b-486b-a448-cc08b9084f44 admin admin] PATH_INFO: `/v3/projects/506f3ab4eec248b38c494a68d7f951f1` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:59:22.416663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6025e502-cd8b-486b-a448-cc08b9084f44 admin admin] RBAC: Authorizing `identity:delete_project(project_id=506f3ab4eec248b38c494a68d7f951f1)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 00:59:22.418828 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6025e502-cd8b-486b-a448-cc08b9084f44 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 00:59:22.501789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-6025e502-cd8b-486b-a448-cc08b9084f44 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '506f3ab4eec248b38c494a68d7f951f1', 'request_id': 'req-6025e502-cd8b-486b-a448-cc08b9084f44'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:59:22.508956 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-6025e502-cd8b-486b-a448-cc08b9084f44 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '506f3ab4eec248b38c494a68d7f951f1', 'request_id': 'req-6025e502-cd8b-486b-a448-cc08b9084f44'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:59:22.510728 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=93,keystone:INSERT=1,keystone:DELETE=6 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:59:22.519885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-6025e502-cd8b-486b-a448-cc08b9084f44 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '506f3ab4eec248b38c494a68d7f951f1', 'request_id': 'req-6025e502-cd8b-486b-a448-cc08b9084f44'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:59:22.533445 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-6025e502-cd8b-486b-a448-cc08b9084f44 admin admin] The token cache is being invalidate because project 506f3ab4eec248b38c494a68d7f951f1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 00:59:22.533738 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-6025e502-cd8b-486b-a448-cc08b9084f44 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:59:22.534371 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-6025e502-cd8b-486b-a448-cc08b9084f44 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 00:59:22.535908 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 911/1832] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 00:59:22 2022] DELETE /identity/v3/projects/506f3ab4eec248b38c494a68d7f951f1 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 00:59:23.613995 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f284ee-323a-42db-83e3-ebba97f6f565 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:59:23.614578 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f284ee-323a-42db-83e3-ebba97f6f565 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:59:23.614578 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f284ee-323a-42db-83e3-ebba97f6f565 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:59:23.635487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-77f284ee-323a-42db-83e3-ebba97f6f565 None None] MFA Rules not processed for user `bc8294d49c654c8f9d8f7539b359df85`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:59:23.636743 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-77f284ee-323a-42db-83e3-ebba97f6f565 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:59:23.662450 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:59:23.698413 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 922/1833] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:59:23 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:59:29.756981 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251991f2-af6f-4e7a-b6ac-b045df3fecde None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 00:59:29.757554 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251991f2-af6f-4e7a-b6ac-b045df3fecde None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 00:59:29.757554 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251991f2-af6f-4e7a-b6ac-b045df3fecde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 00:59:29.777390 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-251991f2-af6f-4e7a-b6ac-b045df3fecde None None] MFA Rules not processed for user `5f198d5e999b487bad75a6b423fe1adc`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 00:59:29.778623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-251991f2-af6f-4e7a-b6ac-b045df3fecde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 00:59:29.834661 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 912/1834] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 00:59:29 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 00:59:33.690644 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=12 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 00:59:39.827886 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:INSERT=1,keystone:DELETE=1,keystone:SELECT=29 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:00:01.872790 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34501602-7537-476d-9e8f-6343d6db6087 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:01.873352 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34501602-7537-476d-9e8f-6343d6db6087 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:01.873352 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34501602-7537-476d-9e8f-6343d6db6087 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:01.893040 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-34501602-7537-476d-9e8f-6343d6db6087 None None] MFA Rules not processed for user `4716e7afe2784e2a9b872996d895c374`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:00:01.894231 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-34501602-7537-476d-9e8f-6343d6db6087 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:01.949745 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 923/1835] 10.222.0.16 () {58 vars in 1009 bytes} [Sun Oct 30 01:00:01 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:00:01.993954 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251991f2-af6f-4e7a-b6ac-b045df3fecde None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:00:01.995528 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-251991f2-af6f-4e7a-b6ac-b045df3fecde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:02.069740 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251991f2-af6f-4e7a-b6ac-b045df3fecde None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:00:02.076075 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ca90ae-2a7e-4afc-8c25-e08b88f2518b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '67f6eff8c0254cc6a210101dd7af988d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:00:02.076679 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ca90ae-2a7e-4afc-8c25-e08b88f2518b service swift] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:02.077209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ca90ae-2a7e-4afc-8c25-e08b88f2518b service swift] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:02.077680 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ca90ae-2a7e-4afc-8c25-e08b88f2518b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:02.086612 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ca90ae-2a7e-4afc-8c25-e08b88f2518b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:00:02.088266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ca90ae-2a7e-4afc-8c25-e08b88f2518b service swift] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:00:02.096403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 913/1836] 10.222.0.16 () {60 vars in 1384 bytes} [Sun Oct 30 01:00:01 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 30 01:00:02.639462 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af957b9-80f9-4af8-98fb-2f2da783cc9f None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:02.639851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af957b9-80f9-4af8-98fb-2f2da783cc9f None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:02.640154 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af957b9-80f9-4af8-98fb-2f2da783cc9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:02.659935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-8af957b9-80f9-4af8-98fb-2f2da783cc9f None None] MFA Rules not processed for user `4716e7afe2784e2a9b872996d895c374`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:00:02.661096 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-8af957b9-80f9-4af8-98fb-2f2da783cc9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:02.668669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 924/1837] 10.222.0.16 () {58 vars in 967 bytes} [Sun Oct 30 01:00:02 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:00:02.682304 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ca90ae-2a7e-4afc-8c25-e08b88f2518b service swift] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:00:02.695072 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ca90ae-2a7e-4afc-8c25-e08b88f2518b service swift] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:00:02.701271 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0bab43-7571-4096-a6e6-0cba9fde1308 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '67f6eff8c0254cc6a210101dd7af988d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:00:02.701802 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0bab43-7571-4096-a6e6-0cba9fde1308 service swift] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:02.702072 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0bab43-7571-4096-a6e6-0cba9fde1308 service swift] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:02.702331 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0bab43-7571-4096-a6e6-0cba9fde1308 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:02.712228 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0bab43-7571-4096-a6e6-0cba9fde1308 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:00:02.713876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0bab43-7571-4096-a6e6-0cba9fde1308 service swift] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:00:02.725279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 914/1838] 10.222.0.16 () {60 vars in 1384 bytes} [Sun Oct 30 01:00:02 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 30 01:00:03.008993 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3045bb48-e03e-41cb-af05-f204b61c16ed None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:03.009502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3045bb48-e03e-41cb-af05-f204b61c16ed None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:03.009502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3045bb48-e03e-41cb-af05-f204b61c16ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:03.029257 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-3045bb48-e03e-41cb-af05-f204b61c16ed None None] MFA Rules not processed for user `4716e7afe2784e2a9b872996d895c374`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:00:03.030450 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3045bb48-e03e-41cb-af05-f204b61c16ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:03.038614 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 925/1839] 10.222.0.16 () {58 vars in 1009 bytes} [Sun Oct 30 01:00:03 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:00:03.051016 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0bab43-7571-4096-a6e6-0cba9fde1308 service swift] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:00:03.062042 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0bab43-7571-4096-a6e6-0cba9fde1308 service swift] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:00:03.068966 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fcb484-5d22-4e73-a812-62f1caca074c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '67f6eff8c0254cc6a210101dd7af988d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:00:03.069805 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fcb484-5d22-4e73-a812-62f1caca074c service swift] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:03.070077 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fcb484-5d22-4e73-a812-62f1caca074c service swift] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:03.070333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fcb484-5d22-4e73-a812-62f1caca074c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:03.080034 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fcb484-5d22-4e73-a812-62f1caca074c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:00:03.082989 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fcb484-5d22-4e73-a812-62f1caca074c service swift] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:00:03.091707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 915/1840] 10.222.0.16 () {60 vars in 1384 bytes} [Sun Oct 30 01:00:03 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 30 01:00:03.422889 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fddce7c-9dde-4e8c-a5d3-ee271b8c65e3 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:03.423223 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fddce7c-9dde-4e8c-a5d3-ee271b8c65e3 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:03.423496 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fddce7c-9dde-4e8c-a5d3-ee271b8c65e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:03.444248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-4fddce7c-9dde-4e8c-a5d3-ee271b8c65e3 None None] MFA Rules not processed for user `4716e7afe2784e2a9b872996d895c374`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:00:03.445437 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-4fddce7c-9dde-4e8c-a5d3-ee271b8c65e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:03.454147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 926/1841] 10.222.0.16 () {58 vars in 967 bytes} [Sun Oct 30 01:00:03 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:00:03.466665 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fcb484-5d22-4e73-a812-62f1caca074c service swift] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:00:03.478029 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fcb484-5d22-4e73-a812-62f1caca074c service swift] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:00:03.484289 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bd55d9-4641-42d0-93c0-407423c46e3f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '67f6eff8c0254cc6a210101dd7af988d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:00:03.485055 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bd55d9-4641-42d0-93c0-407423c46e3f service swift] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:03.485256 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bd55d9-4641-42d0-93c0-407423c46e3f service swift] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:03.485524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bd55d9-4641-42d0-93c0-407423c46e3f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:03.493884 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bd55d9-4641-42d0-93c0-407423c46e3f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:00:03.495453 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bd55d9-4641-42d0-93c0-407423c46e3f service swift] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:00:03.503874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 916/1842] 10.222.0.16 () {60 vars in 1384 bytes} [Sun Oct 30 01:00:03 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 30 01:00:05.687381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fddce7c-9dde-4e8c-a5d3-ee271b8c65e3 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:00:05.689526 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-4fddce7c-9dde-4e8c-a5d3-ee271b8c65e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:05.771642 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fddce7c-9dde-4e8c-a5d3-ee271b8c65e3 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:00:05.776563 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea252b89-521d-46eb-8696-e227f450f25d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:00:05.777206 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea252b89-521d-46eb-8696-e227f450f25d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:05.777643 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea252b89-521d-46eb-8696-e227f450f25d admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:05.777918 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea252b89-521d-46eb-8696-e227f450f25d admin admin] PATH_INFO: `/v3/users/bc8294d49c654c8f9d8f7539b359df85` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:05.779229 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea252b89-521d-46eb-8696-e227f450f25d admin admin] RBAC: Authorizing `identity:delete_user(user_id=bc8294d49c654c8f9d8f7539b359df85)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:00:05.781174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea252b89-521d-46eb-8696-e227f450f25d admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:00:05.830612 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ea252b89-521d-46eb-8696-e227f450f25d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bc8294d49c654c8f9d8f7539b359df85'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:00:05.840230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ea252b89-521d-46eb-8696-e227f450f25d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bc8294d49c654c8f9d8f7539b359df85'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:00:05.840969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ea252b89-521d-46eb-8696-e227f450f25d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bc8294d49c654c8f9d8f7539b359df85'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:00:05.854489 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ea252b89-521d-46eb-8696-e227f450f25d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bc8294d49c654c8f9d8f7539b359df85'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:00:05.861607 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ea252b89-521d-46eb-8696-e227f450f25d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bc8294d49c654c8f9d8f7539b359df85'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:00:05.869144 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 927/1843] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:00:05 2022] DELETE /identity/v3/users/bc8294d49c654c8f9d8f7539b359df85 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:00:05.873514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5378ce1-2af3-4e3a-819f-69945e9b4860 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:05.874053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5378ce1-2af3-4e3a-819f-69945e9b4860 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:05.874053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5378ce1-2af3-4e3a-819f-69945e9b4860 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:05.893494 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-e5378ce1-2af3-4e3a-819f-69945e9b4860 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:00:05.894829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-e5378ce1-2af3-4e3a-819f-69945e9b4860 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:05.964983 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 917/1844] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:00:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:00:05.975026 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea252b89-521d-46eb-8696-e227f450f25d admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:00:05.976616 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ea252b89-521d-46eb-8696-e227f450f25d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:05.986251 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:00:06.066947 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea252b89-521d-46eb-8696-e227f450f25d admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:00:06.070592 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa0e169-57c8-4a3d-ae93-4808d68bec11 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:00:06.071207 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa0e169-57c8-4a3d-ae93-4808d68bec11 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:06.071502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa0e169-57c8-4a3d-ae93-4808d68bec11 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:06.071791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa0e169-57c8-4a3d-ae93-4808d68bec11 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:06.082889 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa0e169-57c8-4a3d-ae93-4808d68bec11 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:00:06.084399 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa0e169-57c8-4a3d-ae93-4808d68bec11 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:00:06.093984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 928/1845] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:00:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:00:06.275476 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5378ce1-2af3-4e3a-819f-69945e9b4860 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:00:06.278434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-e5378ce1-2af3-4e3a-819f-69945e9b4860 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:06.308195 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5378ce1-2af3-4e3a-819f-69945e9b4860 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:00:06.315902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13761fd-f81d-41ec-835e-ac8e7bb7b419 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:00:06.316779 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13761fd-f81d-41ec-835e-ac8e7bb7b419 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:06.317185 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13761fd-f81d-41ec-835e-ac8e7bb7b419 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:06.317582 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13761fd-f81d-41ec-835e-ac8e7bb7b419 admin admin] PATH_INFO: `/v3/projects/27fc853acd7b4ef3803aec20a793bd06` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:06.318948 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13761fd-f81d-41ec-835e-ac8e7bb7b419 admin admin] RBAC: Authorizing `identity:delete_project(project_id=27fc853acd7b4ef3803aec20a793bd06)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:00:06.321051 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13761fd-f81d-41ec-835e-ac8e7bb7b419 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:00:06.376677 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:00:06.380095 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b13761fd-f81d-41ec-835e-ac8e7bb7b419 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '27fc853acd7b4ef3803aec20a793bd06', 'request_id': 'req-b13761fd-f81d-41ec-835e-ac8e7bb7b419'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:00:06.386851 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b13761fd-f81d-41ec-835e-ac8e7bb7b419 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '27fc853acd7b4ef3803aec20a793bd06', 'request_id': 'req-b13761fd-f81d-41ec-835e-ac8e7bb7b419'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:00:06.395814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b13761fd-f81d-41ec-835e-ac8e7bb7b419 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '27fc853acd7b4ef3803aec20a793bd06', 'request_id': 'req-b13761fd-f81d-41ec-835e-ac8e7bb7b419'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:00:06.407130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b13761fd-f81d-41ec-835e-ac8e7bb7b419 admin admin] The token cache is being invalidate because project 27fc853acd7b4ef3803aec20a793bd06 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:00:06.407391 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b13761fd-f81d-41ec-835e-ac8e7bb7b419 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:00:06.408028 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b13761fd-f81d-41ec-835e-ac8e7bb7b419 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:00:06.409526 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 918/1846] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:00:06 2022] DELETE /identity/v3/projects/27fc853acd7b4ef3803aec20a793bd06 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:00:16.089868 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=24 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:00:16.404061 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:DELETE=4,keystone:SELECT=7,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:00:23.688483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33c015e-0382-45ab-bf54-522dec0ffe1c None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:23.689003 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33c015e-0382-45ab-bf54-522dec0ffe1c None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:23.689119 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33c015e-0382-45ab-bf54-522dec0ffe1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:23.718464 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-a33c015e-0382-45ab-bf54-522dec0ffe1c None None] MFA Rules not processed for user `4716e7afe2784e2a9b872996d895c374`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:00:23.720474 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-a33c015e-0382-45ab-bf54-522dec0ffe1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:23.777765 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 929/1847] 10.222.0.16 () {58 vars in 1009 bytes} [Sun Oct 30 01:00:23 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:00:23.790527 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13761fd-f81d-41ec-835e-ac8e7bb7b419 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:00:23.792135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b13761fd-f81d-41ec-835e-ac8e7bb7b419 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:23.863942 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13761fd-f81d-41ec-835e-ac8e7bb7b419 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:00:23.869563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b082d8-8a91-49b9-acec-4572d30c4ec8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '67f6eff8c0254cc6a210101dd7af988d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:00:23.870172 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b082d8-8a91-49b9-acec-4572d30c4ec8 service swift] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:23.870440 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b082d8-8a91-49b9-acec-4572d30c4ec8 service swift] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:23.871113 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b082d8-8a91-49b9-acec-4572d30c4ec8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:23.879420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b082d8-8a91-49b9-acec-4572d30c4ec8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:00:23.881153 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b082d8-8a91-49b9-acec-4572d30c4ec8 service swift] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:00:23.891227 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 919/1848] 10.222.0.16 () {60 vars in 1384 bytes} [Sun Oct 30 01:00:23 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 30 01:00:29.595528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42eb237d-fa38-45df-90bc-f4ffe7ee8f7e None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:29.596256 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42eb237d-fa38-45df-90bc-f4ffe7ee8f7e None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:29.596256 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42eb237d-fa38-45df-90bc-f4ffe7ee8f7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:29.627442 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-42eb237d-fa38-45df-90bc-f4ffe7ee8f7e None None] MFA Rules not processed for user `ddeaa7a90ee7483590e6bc0d189e1e2a`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:00:29.629308 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-42eb237d-fa38-45df-90bc-f4ffe7ee8f7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:29.716592 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 930/1849] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:00:29 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:00:33.888861 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=32 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:00:39.700751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=50 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:00:42.923669 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bac6739-a455-4848-b7ff-cf3f88d83374 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:42.924196 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bac6739-a455-4848-b7ff-cf3f88d83374 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:42.924286 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bac6739-a455-4848-b7ff-cf3f88d83374 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:42.943667 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-3bac6739-a455-4848-b7ff-cf3f88d83374 None None] MFA Rules not processed for user `4716e7afe2784e2a9b872996d895c374`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:00:42.944929 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-3bac6739-a455-4848-b7ff-cf3f88d83374 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:42.952843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 920/1850] 10.222.0.16 () {58 vars in 967 bytes} [Sun Oct 30 01:00:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:00:42.965953 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42eb237d-fa38-45df-90bc-f4ffe7ee8f7e None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:00:42.977614 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42eb237d-fa38-45df-90bc-f4ffe7ee8f7e None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:00:42.982647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1adc5190-0d3d-41ba-870c-60d0d6429b1a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '67f6eff8c0254cc6a210101dd7af988d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:00:42.983261 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adc5190-0d3d-41ba-870c-60d0d6429b1a service swift] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:42.983537 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adc5190-0d3d-41ba-870c-60d0d6429b1a service swift] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:42.983865 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adc5190-0d3d-41ba-870c-60d0d6429b1a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:42.992230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1adc5190-0d3d-41ba-870c-60d0d6429b1a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:00:42.993819 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1adc5190-0d3d-41ba-870c-60d0d6429b1a service swift] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:00:43.002266 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 931/1851] 10.222.0.16 () {60 vars in 1384 bytes} [Sun Oct 30 01:00:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 30 01:00:45.194923 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4975258-6aaf-4f20-b94d-6f9e12dbf797 None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:45.195420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4975258-6aaf-4f20-b94d-6f9e12dbf797 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:45.195613 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4975258-6aaf-4f20-b94d-6f9e12dbf797 None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:45.197186 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 921/1852] 10.222.0.16 () {54 vars in 921 bytes} [Sun Oct 30 01:00:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 01:00:45.226168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184781cb-6751-4b73-a6cb-5cf55c904606 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:45.226686 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184781cb-6751-4b73-a6cb-5cf55c904606 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:45.226886 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184781cb-6751-4b73-a6cb-5cf55c904606 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:45.249756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-184781cb-6751-4b73-a6cb-5cf55c904606 None None] MFA Rules not processed for user `63cc035057064805a95e9f12714ebade`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:00:45.251022 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-184781cb-6751-4b73-a6cb-5cf55c904606 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:45.269314 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4975258-6aaf-4f20-b94d-6f9e12dbf797 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:00:45.272099 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d4975258-6aaf-4f20-b94d-6f9e12dbf797 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:45.332676 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 932/1853] 10.222.0.16 () {58 vars in 1018 bytes} [Sun Oct 30 01:00:45 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:00:45.346996 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184781cb-6751-4b73-a6cb-5cf55c904606 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:00:45.348522 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-184781cb-6751-4b73-a6cb-5cf55c904606 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:45.350328 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4975258-6aaf-4f20-b94d-6f9e12dbf797 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:00:45.355336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b298ac45-656d-459f-bbde-90683f0f9c18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:00:45.356251 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b298ac45-656d-459f-bbde-90683f0f9c18 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:45.356590 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b298ac45-656d-459f-bbde-90683f0f9c18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:45.357116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b298ac45-656d-459f-bbde-90683f0f9c18 admin admin] PATH_INFO: `/v3/users/5f198d5e999b487bad75a6b423fe1adc` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:45.358325 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b298ac45-656d-459f-bbde-90683f0f9c18 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5f198d5e999b487bad75a6b423fe1adc)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:00:45.360435 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b298ac45-656d-459f-bbde-90683f0f9c18 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:00:45.362986 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184781cb-6751-4b73-a6cb-5cf55c904606 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:00:45.371884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0173d3c-7227-4360-aa2c-6f547f956982 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '63cc035057064805a95e9f12714ebade', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:00:45.372735 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0173d3c-7227-4360-aa2c-6f547f956982 service placement] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:45.373188 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0173d3c-7227-4360-aa2c-6f547f956982 service placement] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:45.373606 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0173d3c-7227-4360-aa2c-6f547f956982 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:45.384674 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0173d3c-7227-4360-aa2c-6f547f956982 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:00:45.386400 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0173d3c-7227-4360-aa2c-6f547f956982 service placement] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:00:45.399961 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 933/1854] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:00:45 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 01:00:45.414833 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b298ac45-656d-459f-bbde-90683f0f9c18 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5f198d5e999b487bad75a6b423fe1adc'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:00:45.434581 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b298ac45-656d-459f-bbde-90683f0f9c18 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5f198d5e999b487bad75a6b423fe1adc'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:00:45.435584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b298ac45-656d-459f-bbde-90683f0f9c18 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5f198d5e999b487bad75a6b423fe1adc'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:00:45.443695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b298ac45-656d-459f-bbde-90683f0f9c18 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5f198d5e999b487bad75a6b423fe1adc'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:00:45.455737 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b298ac45-656d-459f-bbde-90683f0f9c18 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5f198d5e999b487bad75a6b423fe1adc'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:00:45.461896 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 922/1855] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:00:45 2022] DELETE /identity/v3/users/5f198d5e999b487bad75a6b423fe1adc => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:00:45.642427 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0173d3c-7227-4360-aa2c-6f547f956982 service placement] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:00:45.653789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0173d3c-7227-4360-aa2c-6f547f956982 service placement] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:00:45.713556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c54c33-29e1-4181-a547-b6483906f055 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:00:45.714209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c54c33-29e1-4181-a547-b6483906f055 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:45.714490 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c54c33-29e1-4181-a547-b6483906f055 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:45.714758 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c54c33-29e1-4181-a547-b6483906f055 admin admin] PATH_INFO: `/v3/projects/dd16277cbc344c9aa8ffbc45ce1a16d2` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:45.715980 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c54c33-29e1-4181-a547-b6483906f055 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dd16277cbc344c9aa8ffbc45ce1a16d2)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:00:45.717892 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c54c33-29e1-4181-a547-b6483906f055 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:00:45.772063 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-92c54c33-29e1-4181-a547-b6483906f055 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dd16277cbc344c9aa8ffbc45ce1a16d2', 'request_id': 'req-92c54c33-29e1-4181-a547-b6483906f055'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:00:45.780628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-92c54c33-29e1-4181-a547-b6483906f055 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dd16277cbc344c9aa8ffbc45ce1a16d2', 'request_id': 'req-92c54c33-29e1-4181-a547-b6483906f055'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:00:45.786547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-92c54c33-29e1-4181-a547-b6483906f055 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dd16277cbc344c9aa8ffbc45ce1a16d2', 'request_id': 'req-92c54c33-29e1-4181-a547-b6483906f055'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:00:45.794453 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:00:45.798208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-92c54c33-29e1-4181-a547-b6483906f055 admin admin] The token cache is being invalidate because project dd16277cbc344c9aa8ffbc45ce1a16d2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:00:45.798485 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-92c54c33-29e1-4181-a547-b6483906f055 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:00:45.799243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-92c54c33-29e1-4181-a547-b6483906f055 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:00:45.801236 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 934/1856] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:00:45 2022] DELETE /identity/v3/projects/dd16277cbc344c9aa8ffbc45ce1a16d2 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:00:45.807322 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe2a73b-903f-490e-a1ad-caa119a05076 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:45.807604 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe2a73b-903f-490e-a1ad-caa119a05076 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:45.807901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe2a73b-903f-490e-a1ad-caa119a05076 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:45.827672 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-6fe2a73b-903f-490e-a1ad-caa119a05076 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:00:45.828887 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-6fe2a73b-903f-490e-a1ad-caa119a05076 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:45.893104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 923/1857] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:00:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:00:45.901152 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf1d700-1ab0-45b5-aad1-3ec0c8beb879 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:45.901431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf1d700-1ab0-45b5-aad1-3ec0c8beb879 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:45.901699 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf1d700-1ab0-45b5-aad1-3ec0c8beb879 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:45.921922 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-faf1d700-1ab0-45b5-aad1-3ec0c8beb879 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:00:45.923108 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-faf1d700-1ab0-45b5-aad1-3ec0c8beb879 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:45.930998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 935/1858] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:00:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:00:45.936195 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe2a73b-903f-490e-a1ad-caa119a05076 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:00:45.946178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe2a73b-903f-490e-a1ad-caa119a05076 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:00:45.951274 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f49bbb-317c-4b7a-b31c-224ceb1d5d3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:00:45.952376 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f49bbb-317c-4b7a-b31c-224ceb1d5d3f admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:45.952783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f49bbb-317c-4b7a-b31c-224ceb1d5d3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:45.953176 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f49bbb-317c-4b7a-b31c-224ceb1d5d3f admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:45.953901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f49bbb-317c-4b7a-b31c-224ceb1d5d3f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:00:45.954199 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f49bbb-317c-4b7a-b31c-224ceb1d5d3f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:00:45.955817 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f49bbb-317c-4b7a-b31c-224ceb1d5d3f admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:00:45.965419 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:00:45.969625 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 924/1859] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:00:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 01:00:45.974021 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf1d700-1ab0-45b5-aad1-3ec0c8beb879 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:00:45.984376 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf1d700-1ab0-45b5-aad1-3ec0c8beb879 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:00:45.989032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b76d32f-25f1-437f-b3e3-95865ee446af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:00:45.989640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b76d32f-25f1-437f-b3e3-95865ee446af admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:45.989911 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b76d32f-25f1-437f-b3e3-95865ee446af admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:45.990174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b76d32f-25f1-437f-b3e3-95865ee446af admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:45.990792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b76d32f-25f1-437f-b3e3-95865ee446af admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:00:45.992748 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b76d32f-25f1-437f-b3e3-95865ee446af admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:00:46.025532 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 936/1860] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:00:45 2022] POST /identity/v3/projects => generated 369 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:00:46.027039 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f49bbb-317c-4b7a-b31c-224ceb1d5d3f admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:00:46.036985 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f49bbb-317c-4b7a-b31c-224ceb1d5d3f admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:00:46.103832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a855dbb9-f94d-4e1a-9cac-0ec26c841b8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:00:46.104475 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a855dbb9-f94d-4e1a-9cac-0ec26c841b8e admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:46.104752 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a855dbb9-f94d-4e1a-9cac-0ec26c841b8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:46.105113 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a855dbb9-f94d-4e1a-9cac-0ec26c841b8e admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:46.105790 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a855dbb9-f94d-4e1a-9cac-0ec26c841b8e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:00:46.107763 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a855dbb9-f94d-4e1a-9cac-0ec26c841b8e admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:00:46.131854 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 925/1861] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:00:46 2022] POST /identity/v3/users => generated 285 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:00:46.135741 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b76d32f-25f1-437f-b3e3-95865ee446af admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:00:46.147580 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b76d32f-25f1-437f-b3e3-95865ee446af admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:00:46.154282 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20604952-8523-4b4e-9efd-c73917c37928 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:00:46.154983 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20604952-8523-4b4e-9efd-c73917c37928 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:46.155301 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20604952-8523-4b4e-9efd-c73917c37928 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:46.155574 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20604952-8523-4b4e-9efd-c73917c37928 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:46.156492 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20604952-8523-4b4e-9efd-c73917c37928 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:00:46.158239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20604952-8523-4b4e-9efd-c73917c37928 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:00:46.169521 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 937/1862] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:00:46 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:00:46.173619 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a855dbb9-f94d-4e1a-9cac-0ec26c841b8e admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:00:46.187277 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a855dbb9-f94d-4e1a-9cac-0ec26c841b8e admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:00:46.193406 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6940a419-5aea-47ec-a668-835c2b2e9fb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:00:46.194027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6940a419-5aea-47ec-a668-835c2b2e9fb2 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:46.194305 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6940a419-5aea-47ec-a668-835c2b2e9fb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:46.194581 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6940a419-5aea-47ec-a668-835c2b2e9fb2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:46.195318 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6940a419-5aea-47ec-a668-835c2b2e9fb2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:00:46.196851 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6940a419-5aea-47ec-a668-835c2b2e9fb2 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:00:46.209041 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 926/1863] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:00:46 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:00:46.213750 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20604952-8523-4b4e-9efd-c73917c37928 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:00:46.223834 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20604952-8523-4b4e-9efd-c73917c37928 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:00:46.228932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dd5cc6-55ca-4ee6-b3dd-0b00199537b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:00:46.229612 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dd5cc6-55ca-4ee6-b3dd-0b00199537b6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:46.229882 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dd5cc6-55ca-4ee6-b3dd-0b00199537b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:46.230151 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dd5cc6-55ca-4ee6-b3dd-0b00199537b6 admin admin] PATH_INFO: `/v3/projects/ac8d8c3f3a234849ba1bd7d47e38ad71/users/65563eaa414f4967a3303d2e78b0cab7/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:46.248398 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59dd5cc6-55ca-4ee6-b3dd-0b00199537b6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac8d8c3f3a234849ba1bd7d47e38ad71, user_id=65563eaa414f4967a3303d2e78b0cab7, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:00:46.251062 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59dd5cc6-55ca-4ee6-b3dd-0b00199537b6 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:00:46.266585 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 938/1864] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:00:46 2022] PUT /identity/v3/projects/ac8d8c3f3a234849ba1bd7d47e38ad71/users/65563eaa414f4967a3303d2e78b0cab7/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:00:46.336733 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b0e365-7c6e-42e8-94ad-ff5b7524cb90 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:46.337007 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b0e365-7c6e-42e8-94ad-ff5b7524cb90 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:46.337340 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b0e365-7c6e-42e8-94ad-ff5b7524cb90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:46.368299 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-37b0e365-7c6e-42e8-94ad-ff5b7524cb90 None None] MFA Rules not processed for user `65563eaa414f4967a3303d2e78b0cab7`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:00:46.369520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-37b0e365-7c6e-42e8-94ad-ff5b7524cb90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:46.437320 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 927/1865] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:00:46 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:00:46.450044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dd5cc6-55ca-4ee6-b3dd-0b00199537b6 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:00:46.452492 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-59dd5cc6-55ca-4ee6-b3dd-0b00199537b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:46.511687 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dd5cc6-55ca-4ee6-b3dd-0b00199537b6 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:00:46.515651 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a99dded-abe6-426f-8ed1-7c29835cf36b service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '807b123652d245269336fcffaf36478e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:00:46.516294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a99dded-abe6-426f-8ed1-7c29835cf36b service glance] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:46.516568 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a99dded-abe6-426f-8ed1-7c29835cf36b service glance] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:46.516838 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a99dded-abe6-426f-8ed1-7c29835cf36b service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:46.525726 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a99dded-abe6-426f-8ed1-7c29835cf36b service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:00:46.527321 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a99dded-abe6-426f-8ed1-7c29835cf36b service glance] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:00:46.536598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 939/1866] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:00:46 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:00:48.399916 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b0e365-7c6e-42e8-94ad-ff5b7524cb90 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:00:48.401823 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-37b0e365-7c6e-42e8-94ad-ff5b7524cb90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:48.464589 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b0e365-7c6e-42e8-94ad-ff5b7524cb90 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:00:48.468516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639a4e66-6997-4c89-ac74-19b30e7fccc3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:00:48.469177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639a4e66-6997-4c89-ac74-19b30e7fccc3 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:48.469385 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639a4e66-6997-4c89-ac74-19b30e7fccc3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:48.469642 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639a4e66-6997-4c89-ac74-19b30e7fccc3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:48.471337 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-639a4e66-6997-4c89-ac74-19b30e7fccc3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:48.493330 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:00:48.547532 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639a4e66-6997-4c89-ac74-19b30e7fccc3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:00:48.549021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639a4e66-6997-4c89-ac74-19b30e7fccc3 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:00:48.560292 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 928/1867] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:00:48 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:00:50.081254 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a99dded-abe6-426f-8ed1-7c29835cf36b service glance] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:00:50.091012 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a99dded-abe6-426f-8ed1-7c29835cf36b service glance] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:00:50.094865 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782c4d8d-dab3-41e1-89a1-88c880ff0a67 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:00:50.095481 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782c4d8d-dab3-41e1-89a1-88c880ff0a67 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:50.095931 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782c4d8d-dab3-41e1-89a1-88c880ff0a67 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:50.096238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782c4d8d-dab3-41e1-89a1-88c880ff0a67 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:50.098078 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-782c4d8d-dab3-41e1-89a1-88c880ff0a67 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:50.111518 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782c4d8d-dab3-41e1-89a1-88c880ff0a67 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:00:50.112986 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782c4d8d-dab3-41e1-89a1-88c880ff0a67 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:00:50.124691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 940/1868] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:00:50 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:00:52.091014 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daeaeb0d-efdf-4940-ba2f-e65b64ac40a8 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:52.091552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daeaeb0d-efdf-4940-ba2f-e65b64ac40a8 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:52.091552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daeaeb0d-efdf-4940-ba2f-e65b64ac40a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:52.128242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-daeaeb0d-efdf-4940-ba2f-e65b64ac40a8 None None] MFA Rules not processed for user `4716e7afe2784e2a9b872996d895c374`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:00:52.129959 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-daeaeb0d-efdf-4940-ba2f-e65b64ac40a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:52.204472 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 929/1869] 10.222.0.16 () {58 vars in 1009 bytes} [Sun Oct 30 01:00:52 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:00:52.218405 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782c4d8d-dab3-41e1-89a1-88c880ff0a67 service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:00:52.219861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-782c4d8d-dab3-41e1-89a1-88c880ff0a67 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:52.312206 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782c4d8d-dab3-41e1-89a1-88c880ff0a67 service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:00:52.318799 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ade1b1-9d80-420a-b543-a2b5c9349fbf service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '67f6eff8c0254cc6a210101dd7af988d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:00:52.320316 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ade1b1-9d80-420a-b543-a2b5c9349fbf service swift] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:52.320680 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ade1b1-9d80-420a-b543-a2b5c9349fbf service swift] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:52.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ade1b1-9d80-420a-b543-a2b5c9349fbf service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:52.341063 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ade1b1-9d80-420a-b543-a2b5c9349fbf service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:00:52.343539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ade1b1-9d80-420a-b543-a2b5c9349fbf service swift] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:00:52.358625 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 941/1870] 10.222.0.16 () {60 vars in 1384 bytes} [Sun Oct 30 01:00:52 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 30 01:00:53.194689 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82feb7c2-f604-4905-b209-5842d148d2fa None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:53.195233 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82feb7c2-f604-4905-b209-5842d148d2fa None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:53.195397 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82feb7c2-f604-4905-b209-5842d148d2fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:53.225330 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-82feb7c2-f604-4905-b209-5842d148d2fa None None] MFA Rules not processed for user `ecaf8ec105c04b919256f852da859eba`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:00:53.226839 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-82feb7c2-f604-4905-b209-5842d148d2fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:53.359789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 930/1871] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:00:53 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:00:53.638921 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb41ef63-617c-483e-abef-9bb7f182d6ae None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:00:53.639479 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb41ef63-617c-483e-abef-9bb7f182d6ae None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:00:53.639479 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb41ef63-617c-483e-abef-9bb7f182d6ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:00:53.673650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-eb41ef63-617c-483e-abef-9bb7f182d6ae None None] MFA Rules not processed for user `31b7342ecf07408cab6bb040f361fa1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:00:53.675555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-eb41ef63-617c-483e-abef-9bb7f182d6ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:00:53.690959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:00:53.771596 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 942/1872] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:00:53 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:01:00.029174 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82feb7c2-f604-4905-b209-5842d148d2fa None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:01:00.033806 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-82feb7c2-f604-4905-b209-5842d148d2fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:01:00.052000 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82feb7c2-f604-4905-b209-5842d148d2fa None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:01:00.056104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c79675-4909-452e-890c-7e05bc9b16c5 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '807b123652d245269336fcffaf36478e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:01:00.056805 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c79675-4909-452e-890c-7e05bc9b16c5 service glance] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:00.057223 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c79675-4909-452e-890c-7e05bc9b16c5 service glance] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:00.057557 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c79675-4909-452e-890c-7e05bc9b16c5 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:00.059544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-46c79675-4909-452e-890c-7e05bc9b16c5 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:01:00.072373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c79675-4909-452e-890c-7e05bc9b16c5 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:01:00.074085 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c79675-4909-452e-890c-7e05bc9b16c5 service glance] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:01:00.086155 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 931/1873] 10.222.0.16 () {60 vars in 1396 bytes} [Sun Oct 30 01:01:00 2022] GET /identity/v3/auth/tokens?allow_expired=1 => generated 3260 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:01:01.405333 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb41ef63-617c-483e-abef-9bb7f182d6ae None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:01:01.406913 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-eb41ef63-617c-483e-abef-9bb7f182d6ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:01:01.431506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb41ef63-617c-483e-abef-9bb7f182d6ae None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:01:01.438290 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8f61bb-3791-4495-975b-3fa14f46f1d5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:01:01.439210 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8f61bb-3791-4495-975b-3fa14f46f1d5 service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:01.456068 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8f61bb-3791-4495-975b-3fa14f46f1d5 service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:01.456397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8f61bb-3791-4495-975b-3fa14f46f1d5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:01.462996 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-7f8f61bb-3791-4495-975b-3fa14f46f1d5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:01:01.627568 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8f61bb-3791-4495-975b-3fa14f46f1d5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:01:01.629811 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8f61bb-3791-4495-975b-3fa14f46f1d5 service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:01:01.654353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 943/1874] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:01:01 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 249 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:01:01.783904 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c79675-4909-452e-890c-7e05bc9b16c5 service glance] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:01:01.787305 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-46c79675-4909-452e-890c-7e05bc9b16c5 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:01:01.819972 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c79675-4909-452e-890c-7e05bc9b16c5 service glance] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:01:01.829623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee99db5-b53d-4eb0-9fcd-cf0e2d2b3205 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:01:01.831646 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee99db5-b53d-4eb0-9fcd-cf0e2d2b3205 service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:01.832218 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee99db5-b53d-4eb0-9fcd-cf0e2d2b3205 service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:01.832698 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee99db5-b53d-4eb0-9fcd-cf0e2d2b3205 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:01.837818 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-4ee99db5-b53d-4eb0-9fcd-cf0e2d2b3205 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:01:01.985721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:01:02.007166 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee99db5-b53d-4eb0-9fcd-cf0e2d2b3205 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:01:02.009408 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee99db5-b53d-4eb0-9fcd-cf0e2d2b3205 service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:01:02.025740 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 932/1875] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:01:01 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 243 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:01:11.644004 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=46 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:01:12.016306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=7 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:01:12.276542 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8f61bb-3791-4495-975b-3fa14f46f1d5 service nova] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:01:12.279218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-7f8f61bb-3791-4495-975b-3fa14f46f1d5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:01:12.281835 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee99db5-b53d-4eb0-9fcd-cf0e2d2b3205 service nova] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:01:12.284448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-4ee99db5-b53d-4eb0-9fcd-cf0e2d2b3205 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:01:12.298032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8f61bb-3791-4495-975b-3fa14f46f1d5 service nova] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:01:12.300678 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee99db5-b53d-4eb0-9fcd-cf0e2d2b3205 service nova] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:01:12.302124 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a703dd-b2f7-4ab3-9732-6bc9044adc28 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:01:12.302877 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a703dd-b2f7-4ab3-9732-6bc9044adc28 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:12.303216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a703dd-b2f7-4ab3-9732-6bc9044adc28 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:12.303545 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a703dd-b2f7-4ab3-9732-6bc9044adc28 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:12.305602 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee4a799-dcbb-4eeb-911d-5c47eb515af5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:01:12.305602 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee4a799-dcbb-4eeb-911d-5c47eb515af5 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:12.305602 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee4a799-dcbb-4eeb-911d-5c47eb515af5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:12.305913 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-08a703dd-b2f7-4ab3-9732-6bc9044adc28 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:01:12.306018 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee4a799-dcbb-4eeb-911d-5c47eb515af5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:12.308032 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-eee4a799-dcbb-4eeb-911d-5c47eb515af5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:01:12.317396 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a703dd-b2f7-4ab3-9732-6bc9044adc28 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:01:12.318919 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a703dd-b2f7-4ab3-9732-6bc9044adc28 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:01:12.319196 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee4a799-dcbb-4eeb-911d-5c47eb515af5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:01:12.320741 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee4a799-dcbb-4eeb-911d-5c47eb515af5 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:01:12.328419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 944/1876] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:01:12 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:01:12.330182 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 933/1877] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:01:12 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:01:13.759783 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a703dd-b2f7-4ab3-9732-6bc9044adc28 service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:01:13.769183 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a703dd-b2f7-4ab3-9732-6bc9044adc28 service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:01:13.774146 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c663685-9abe-47f8-9bbf-ca3296da963c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:01:13.774751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c663685-9abe-47f8-9bbf-ca3296da963c service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:13.775024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c663685-9abe-47f8-9bbf-ca3296da963c service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:13.775287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c663685-9abe-47f8-9bbf-ca3296da963c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:13.777403 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-6c663685-9abe-47f8-9bbf-ca3296da963c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:01:13.787798 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c663685-9abe-47f8-9bbf-ca3296da963c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:01:13.789358 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c663685-9abe-47f8-9bbf-ca3296da963c service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:01:13.798196 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 945/1878] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:01:13 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:01:13.929520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839f9e71-ce8f-4f12-afb2-0eaa5633586f None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:13.930109 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839f9e71-ce8f-4f12-afb2-0eaa5633586f None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:13.930109 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839f9e71-ce8f-4f12-afb2-0eaa5633586f None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:13.931419 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 934/1879] 10.222.0.16 () {54 vars in 921 bytes} [Sun Oct 30 01:01:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 30 01:01:13.937604 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3faef83a-382f-403d-8101-510039be315d None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:13.937888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3faef83a-382f-403d-8101-510039be315d None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:13.938161 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3faef83a-382f-403d-8101-510039be315d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:13.959884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-3faef83a-382f-403d-8101-510039be315d None None] MFA Rules not processed for user `e7a886715b5b4c1d99f0b37c58cb5417`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:01:13.961091 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3faef83a-382f-403d-8101-510039be315d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:01:13.971092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 946/1880] 10.222.0.16 () {58 vars in 1018 bytes} [Sun Oct 30 01:01:13 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:01:13.982111 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839f9e71-ce8f-4f12-afb2-0eaa5633586f None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:01:13.993504 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839f9e71-ce8f-4f12-afb2-0eaa5633586f None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:01:13.998060 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a2590e-9ecc-48ec-b4a3-63bf42af0b73 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '807b123652d245269336fcffaf36478e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:01:13.998763 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a2590e-9ecc-48ec-b4a3-63bf42af0b73 service glance] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:13.999134 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a2590e-9ecc-48ec-b4a3-63bf42af0b73 service glance] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:13.999455 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a2590e-9ecc-48ec-b4a3-63bf42af0b73 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:14.011981 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a2590e-9ecc-48ec-b4a3-63bf42af0b73 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:01:14.013768 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a2590e-9ecc-48ec-b4a3-63bf42af0b73 service glance] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:01:14.025142 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 935/1881] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:01:13 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:01:14.101882 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620c1150-ac18-4764-961a-8dbd0f5a942b None None] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:14.102182 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620c1150-ac18-4764-961a-8dbd0f5a942b None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:14.102439 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620c1150-ac18-4764-961a-8dbd0f5a942b None None] PATH_INFO: `/` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:14.103852 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 947/1882] 10.222.0.16 () {54 vars in 921 bytes} [Sun Oct 30 01:01:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 01:01:14.108824 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338e9a81-47e4-46da-b166-0b24b86f66a3 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:14.109100 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338e9a81-47e4-46da-b166-0b24b86f66a3 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:14.109371 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338e9a81-47e4-46da-b166-0b24b86f66a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:14.130671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-338e9a81-47e4-46da-b166-0b24b86f66a3 None None] MFA Rules not processed for user `81999329edfb4d4cac567d88d1f52909`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:01:14.131923 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-338e9a81-47e4-46da-b166-0b24b86f66a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:01:14.140027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 936/1883] 10.222.0.16 () {58 vars in 1018 bytes} [Sun Oct 30 01:01:14 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:01:14.152068 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620c1150-ac18-4764-961a-8dbd0f5a942b None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:01:14.164702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620c1150-ac18-4764-961a-8dbd0f5a942b None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:01:14.168552 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0066e60-aae1-4b58-9705-fb9e29c67ff3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:01:14.169245 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0066e60-aae1-4b58-9705-fb9e29c67ff3 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:14.169517 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0066e60-aae1-4b58-9705-fb9e29c67ff3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:14.169797 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0066e60-aae1-4b58-9705-fb9e29c67ff3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:14.176953 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0066e60-aae1-4b58-9705-fb9e29c67ff3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:01:14.178373 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0066e60-aae1-4b58-9705-fb9e29c67ff3 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:01:14.185920 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 948/1884] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:01:14 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:01:14.274142 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b9e1b4-d37d-4eea-aa97-a2021bc08d1a None None] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:14.274421 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b9e1b4-d37d-4eea-aa97-a2021bc08d1a None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:14.274689 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b9e1b4-d37d-4eea-aa97-a2021bc08d1a None None] PATH_INFO: `/` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:14.276126 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 937/1885] 10.222.0.16 () {54 vars in 921 bytes} [Sun Oct 30 01:01:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 30 01:01:14.281946 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e509935-f37b-45c9-b3c1-3ec44c57ebc6 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:14.282226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e509935-f37b-45c9-b3c1-3ec44c57ebc6 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:14.282475 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e509935-f37b-45c9-b3c1-3ec44c57ebc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:14.303783 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-2e509935-f37b-45c9-b3c1-3ec44c57ebc6 None None] MFA Rules not processed for user `e7a886715b5b4c1d99f0b37c58cb5417`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:01:14.304987 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-2e509935-f37b-45c9-b3c1-3ec44c57ebc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:01:14.319411 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 949/1886] 10.222.0.16 () {58 vars in 1018 bytes} [Sun Oct 30 01:01:14 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:01:14.332224 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b9e1b4-d37d-4eea-aa97-a2021bc08d1a None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:01:14.343999 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b9e1b4-d37d-4eea-aa97-a2021bc08d1a None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:01:14.348041 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8138e4e4-892f-4c23-b9f0-94711c2a7bde service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:01:14.349416 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8138e4e4-892f-4c23-b9f0-94711c2a7bde service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:14.349416 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8138e4e4-892f-4c23-b9f0-94711c2a7bde service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:14.349718 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8138e4e4-892f-4c23-b9f0-94711c2a7bde service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:14.360947 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8138e4e4-892f-4c23-b9f0-94711c2a7bde service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:01:14.362379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8138e4e4-892f-4c23-b9f0-94711c2a7bde service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:01:14.373881 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 938/1887] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:01:14 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 01:01:14.436663 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e509935-f37b-45c9-b3c1-3ec44c57ebc6 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:01:14.438164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-2e509935-f37b-45c9-b3c1-3ec44c57ebc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:01:14.506314 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e509935-f37b-45c9-b3c1-3ec44c57ebc6 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:01:14.511304 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e69b534-0f24-4787-93e0-698b250b8f36 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '63cc035057064805a95e9f12714ebade', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:01:14.511952 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e69b534-0f24-4787-93e0-698b250b8f36 service placement] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:14.512233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e69b534-0f24-4787-93e0-698b250b8f36 service placement] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:14.512500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e69b534-0f24-4787-93e0-698b250b8f36 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:14.514412 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-1e69b534-0f24-4787-93e0-698b250b8f36 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:01:14.526098 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e69b534-0f24-4787-93e0-698b250b8f36 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:01:14.527737 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e69b534-0f24-4787-93e0-698b250b8f36 service placement] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:01:14.538329 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 950/1888] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:01:14 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:01:24.369280 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=21 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:01:24.534288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=56 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:01:44.299460 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8138e4e4-892f-4c23-b9f0-94711c2a7bde service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:01:44.310965 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8138e4e4-892f-4c23-b9f0-94711c2a7bde service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:01:44.317042 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba800925-61f0-4b53-bb89-6a931a7fa54e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:01:44.317659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba800925-61f0-4b53-bb89-6a931a7fa54e service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:44.318145 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba800925-61f0-4b53-bb89-6a931a7fa54e service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:44.318566 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba800925-61f0-4b53-bb89-6a931a7fa54e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:44.320573 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-ba800925-61f0-4b53-bb89-6a931a7fa54e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:01:44.339874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba800925-61f0-4b53-bb89-6a931a7fa54e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:01:44.341712 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba800925-61f0-4b53-bb89-6a931a7fa54e service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:01:44.356574 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 939/1889] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:01:44 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:01:53.782953 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de93b93-4f39-4168-b9aa-b237e336f6a4 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:53.783509 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de93b93-4f39-4168-b9aa-b237e336f6a4 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:53.783509 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de93b93-4f39-4168-b9aa-b237e336f6a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:53.812718 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-1de93b93-4f39-4168-b9aa-b237e336f6a4 None None] MFA Rules not processed for user `4716e7afe2784e2a9b872996d895c374`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:01:53.813971 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-1de93b93-4f39-4168-b9aa-b237e336f6a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:01:53.822481 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 951/1890] 10.222.0.16 () {58 vars in 1009 bytes} [Sun Oct 30 01:01:53 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:01:53.857070 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba800925-61f0-4b53-bb89-6a931a7fa54e service nova] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:01:53.872240 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de93b93-4f39-4168-b9aa-b237e336f6a4 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:01:53.879179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba800925-61f0-4b53-bb89-6a931a7fa54e service nova] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:01:53.889963 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de93b93-4f39-4168-b9aa-b237e336f6a4 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:01:53.890524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9414151-d006-4330-9d68-37861300e4de service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:01:53.891158 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9414151-d006-4330-9d68-37861300e4de service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:53.891433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9414151-d006-4330-9d68-37861300e4de service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:53.891695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9414151-d006-4330-9d68-37861300e4de service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:53.893644 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d9414151-d006-4330-9d68-37861300e4de service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:01:53.898879 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24bee33-c532-4ac3-9865-7c98bc4f0aff service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '67f6eff8c0254cc6a210101dd7af988d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:01:53.900970 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24bee33-c532-4ac3-9865-7c98bc4f0aff service swift] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:53.901655 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24bee33-c532-4ac3-9865-7c98bc4f0aff service swift] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:53.902164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24bee33-c532-4ac3-9865-7c98bc4f0aff service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:53.911580 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9414151-d006-4330-9d68-37861300e4de service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:01:53.913961 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9414151-d006-4330-9d68-37861300e4de service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:01:53.914396 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24bee33-c532-4ac3-9865-7c98bc4f0aff service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:01:53.916375 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24bee33-c532-4ac3-9865-7c98bc4f0aff service swift] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:01:53.926558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 952/1891] 10.222.0.16 () {60 vars in 1384 bytes} [Sun Oct 30 01:01:53 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 30 01:01:53.930171 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 940/1892] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:01:53 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 74 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:01:56.721449 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24bee33-c532-4ac3-9865-7c98bc4f0aff service swift] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:01:56.795545 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24bee33-c532-4ac3-9865-7c98bc4f0aff service swift] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:01:56.799975 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdc46dc-81ad-4ad3-993f-ba533bad5178 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:01:56.800600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdc46dc-81ad-4ad3-993f-ba533bad5178 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:56.800873 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdc46dc-81ad-4ad3-993f-ba533bad5178 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:56.801136 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdc46dc-81ad-4ad3-993f-ba533bad5178 admin admin] PATH_INFO: `/v3/users/65563eaa414f4967a3303d2e78b0cab7` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:56.802215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bdc46dc-81ad-4ad3-993f-ba533bad5178 admin admin] RBAC: Authorizing `identity:delete_user(user_id=65563eaa414f4967a3303d2e78b0cab7)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:01:56.804239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bdc46dc-81ad-4ad3-993f-ba533bad5178 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:01:56.862721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-3bdc46dc-81ad-4ad3-993f-ba533bad5178 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '65563eaa414f4967a3303d2e78b0cab7'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:01:56.873913 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-3bdc46dc-81ad-4ad3-993f-ba533bad5178 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '65563eaa414f4967a3303d2e78b0cab7'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:01:56.874673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-3bdc46dc-81ad-4ad3-993f-ba533bad5178 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '65563eaa414f4967a3303d2e78b0cab7'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:01:56.888265 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-3bdc46dc-81ad-4ad3-993f-ba533bad5178 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '65563eaa414f4967a3303d2e78b0cab7'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:01:56.895323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-3bdc46dc-81ad-4ad3-993f-ba533bad5178 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '65563eaa414f4967a3303d2e78b0cab7'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:01:56.901325 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 953/1893] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:01:56 2022] DELETE /identity/v3/users/65563eaa414f4967a3303d2e78b0cab7 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:01:56.912235 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44902759-64d7-4b72-acae-2e9c7f3ecb7b None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:56.913034 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44902759-64d7-4b72-acae-2e9c7f3ecb7b None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:56.913034 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44902759-64d7-4b72-acae-2e9c7f3ecb7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:56.942724 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-44902759-64d7-4b72-acae-2e9c7f3ecb7b None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:01:56.944351 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-44902759-64d7-4b72-acae-2e9c7f3ecb7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:01:57.038338 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 941/1894] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:01:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:01:57.048362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdc46dc-81ad-4ad3-993f-ba533bad5178 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:01:57.049837 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3bdc46dc-81ad-4ad3-993f-ba533bad5178 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:01:57.109813 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdc46dc-81ad-4ad3-993f-ba533bad5178 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:01:57.114860 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc2651a-1cf8-4ed2-82ad-4555f72090e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:01:57.115977 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc2651a-1cf8-4ed2-82ad-4555f72090e8 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:57.116261 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc2651a-1cf8-4ed2-82ad-4555f72090e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:57.116526 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc2651a-1cf8-4ed2-82ad-4555f72090e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:57.128875 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc2651a-1cf8-4ed2-82ad-4555f72090e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:01:57.131183 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc2651a-1cf8-4ed2-82ad-4555f72090e8 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:01:57.143342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 954/1895] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:01:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:01:57.450824 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44902759-64d7-4b72-acae-2e9c7f3ecb7b None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:01:57.452624 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-44902759-64d7-4b72-acae-2e9c7f3ecb7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:01:57.465565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44902759-64d7-4b72-acae-2e9c7f3ecb7b None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:01:57.472045 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655d2d31-1944-4f12-82ce-dcb01294cae5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:01:57.472685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655d2d31-1944-4f12-82ce-dcb01294cae5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:01:57.472957 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655d2d31-1944-4f12-82ce-dcb01294cae5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:01:57.473220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655d2d31-1944-4f12-82ce-dcb01294cae5 admin admin] PATH_INFO: `/v3/projects/ac8d8c3f3a234849ba1bd7d47e38ad71` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:01:57.474504 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655d2d31-1944-4f12-82ce-dcb01294cae5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ac8d8c3f3a234849ba1bd7d47e38ad71)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:01:57.476749 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655d2d31-1944-4f12-82ce-dcb01294cae5 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:01:57.551427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-655d2d31-1944-4f12-82ce-dcb01294cae5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ac8d8c3f3a234849ba1bd7d47e38ad71', 'request_id': 'req-655d2d31-1944-4f12-82ce-dcb01294cae5'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:01:57.557510 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-655d2d31-1944-4f12-82ce-dcb01294cae5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ac8d8c3f3a234849ba1bd7d47e38ad71', 'request_id': 'req-655d2d31-1944-4f12-82ce-dcb01294cae5'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:01:57.566715 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-655d2d31-1944-4f12-82ce-dcb01294cae5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ac8d8c3f3a234849ba1bd7d47e38ad71', 'request_id': 'req-655d2d31-1944-4f12-82ce-dcb01294cae5'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:01:57.579076 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-655d2d31-1944-4f12-82ce-dcb01294cae5 admin admin] The token cache is being invalidate because project ac8d8c3f3a234849ba1bd7d47e38ad71 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:01:57.579703 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-655d2d31-1944-4f12-82ce-dcb01294cae5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:01:57.580449 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-655d2d31-1944-4f12-82ce-dcb01294cae5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:01:57.581949 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 942/1896] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:01:57 2022] DELETE /identity/v3/projects/ac8d8c3f3a234849ba1bd7d47e38ad71 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:02:07.139035 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:02:07.576347 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=66,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:02:22.849614 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc2651a-1cf8-4ed2-82ad-4555f72090e8 service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:02:22.852084 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-cbc2651a-1cf8-4ed2-82ad-4555f72090e8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:02:22.929315 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc2651a-1cf8-4ed2-82ad-4555f72090e8 service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:02:22.996859 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8cccf0-dfb4-4d9b-aa8e-922a46b2164c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:02:22.997471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8cccf0-dfb4-4d9b-aa8e-922a46b2164c service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:22.997748 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8cccf0-dfb4-4d9b-aa8e-922a46b2164c service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:22.998015 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8cccf0-dfb4-4d9b-aa8e-922a46b2164c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:23.000699 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-1f8cccf0-dfb4-4d9b-aa8e-922a46b2164c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:02:23.018186 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8cccf0-dfb4-4d9b-aa8e-922a46b2164c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:02:23.019780 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8cccf0-dfb4-4d9b-aa8e-922a46b2164c service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:02:23.030704 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 955/1897] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:02:22 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:02:30.831242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124ffbfb-71f3-4a8a-90a0-549aa3b766f1 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:30.831782 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124ffbfb-71f3-4a8a-90a0-549aa3b766f1 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:30.831886 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124ffbfb-71f3-4a8a-90a0-549aa3b766f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:30.852438 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-124ffbfb-71f3-4a8a-90a0-549aa3b766f1 None None] MFA Rules not processed for user `4716e7afe2784e2a9b872996d895c374`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:02:30.853523 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-124ffbfb-71f3-4a8a-90a0-549aa3b766f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:02:30.909533 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 943/1898] 10.222.0.16 () {58 vars in 967 bytes} [Sun Oct 30 01:02:30 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:02:30.931560 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8cccf0-dfb4-4d9b-aa8e-922a46b2164c service nova] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:02:30.933242 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-1f8cccf0-dfb4-4d9b-aa8e-922a46b2164c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:02:31.004951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8cccf0-dfb4-4d9b-aa8e-922a46b2164c service nova] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:02:31.009996 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb09295-2877-4c9a-9981-1484d55bbad8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '67f6eff8c0254cc6a210101dd7af988d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:02:31.010644 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb09295-2877-4c9a-9981-1484d55bbad8 service swift] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:31.010919 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb09295-2877-4c9a-9981-1484d55bbad8 service swift] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:31.011173 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb09295-2877-4c9a-9981-1484d55bbad8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:31.022572 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb09295-2877-4c9a-9981-1484d55bbad8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:02:31.024433 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb09295-2877-4c9a-9981-1484d55bbad8 service swift] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:02:31.036092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 956/1899] 10.222.0.16 () {60 vars in 1384 bytes} [Sun Oct 30 01:02:30 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 30 01:02:40.901920 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=25 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:02:40.994518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124ffbfb-71f3-4a8a-90a0-549aa3b766f1 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:02:40.996482 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-124ffbfb-71f3-4a8a-90a0-549aa3b766f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:02:41.032011 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=94 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:02:41.068644 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124ffbfb-71f3-4a8a-90a0-549aa3b766f1 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:02:41.073765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a743bc6-45ed-4288-9a74-1220288953b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:02:41.074401 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a743bc6-45ed-4288-9a74-1220288953b6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:41.074697 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a743bc6-45ed-4288-9a74-1220288953b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:41.074962 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a743bc6-45ed-4288-9a74-1220288953b6 admin admin] PATH_INFO: `/v3/users/ddeaa7a90ee7483590e6bc0d189e1e2a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:41.076176 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a743bc6-45ed-4288-9a74-1220288953b6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ddeaa7a90ee7483590e6bc0d189e1e2a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:02:41.078108 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a743bc6-45ed-4288-9a74-1220288953b6 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:02:41.128425 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4a743bc6-45ed-4288-9a74-1220288953b6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ddeaa7a90ee7483590e6bc0d189e1e2a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:02:41.137645 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4a743bc6-45ed-4288-9a74-1220288953b6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ddeaa7a90ee7483590e6bc0d189e1e2a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:02:41.138406 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4a743bc6-45ed-4288-9a74-1220288953b6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ddeaa7a90ee7483590e6bc0d189e1e2a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:02:41.146068 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4a743bc6-45ed-4288-9a74-1220288953b6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ddeaa7a90ee7483590e6bc0d189e1e2a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:02:41.158004 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4a743bc6-45ed-4288-9a74-1220288953b6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ddeaa7a90ee7483590e6bc0d189e1e2a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:02:41.163823 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 944/1900] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:02:40 2022] DELETE /identity/v3/users/ddeaa7a90ee7483590e6bc0d189e1e2a => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:02:41.337515 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb09295-2877-4c9a-9981-1484d55bbad8 service swift] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:02:41.349586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb09295-2877-4c9a-9981-1484d55bbad8 service swift] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:02:41.411225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e68505-b0a0-47a7-a89a-b9f610602ac0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:02:41.411807 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e68505-b0a0-47a7-a89a-b9f610602ac0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:41.412080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e68505-b0a0-47a7-a89a-b9f610602ac0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:41.412350 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e68505-b0a0-47a7-a89a-b9f610602ac0 admin admin] PATH_INFO: `/v3/projects/2fbc73a853584aaba6109090bd6b1281` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:41.413482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e68505-b0a0-47a7-a89a-b9f610602ac0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2fbc73a853584aaba6109090bd6b1281)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:02:41.415391 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e68505-b0a0-47a7-a89a-b9f610602ac0 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:02:41.466464 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-26e68505-b0a0-47a7-a89a-b9f610602ac0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2fbc73a853584aaba6109090bd6b1281', 'request_id': 'req-26e68505-b0a0-47a7-a89a-b9f610602ac0'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:02:41.479129 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-26e68505-b0a0-47a7-a89a-b9f610602ac0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2fbc73a853584aaba6109090bd6b1281', 'request_id': 'req-26e68505-b0a0-47a7-a89a-b9f610602ac0'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:02:41.485351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-26e68505-b0a0-47a7-a89a-b9f610602ac0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2fbc73a853584aaba6109090bd6b1281', 'request_id': 'req-26e68505-b0a0-47a7-a89a-b9f610602ac0'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:02:41.496425 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-26e68505-b0a0-47a7-a89a-b9f610602ac0 admin admin] The token cache is being invalidate because project 2fbc73a853584aaba6109090bd6b1281 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:02:41.496705 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-26e68505-b0a0-47a7-a89a-b9f610602ac0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:02:41.497355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-26e68505-b0a0-47a7-a89a-b9f610602ac0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:02:41.498853 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 957/1901] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:02:41 2022] DELETE /identity/v3/projects/2fbc73a853584aaba6109090bd6b1281 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:02:41.505161 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6217ac46-1e79-49fe-8325-ae7e16bf3404 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:41.505445 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6217ac46-1e79-49fe-8325-ae7e16bf3404 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:41.505734 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6217ac46-1e79-49fe-8325-ae7e16bf3404 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:41.527242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-6217ac46-1e79-49fe-8325-ae7e16bf3404 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:02:41.528517 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-6217ac46-1e79-49fe-8325-ae7e16bf3404 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:02:41.590282 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 945/1902] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:02:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:02:41.598087 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a104b0-2437-4662-875e-13fb5df1476b None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:41.598372 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a104b0-2437-4662-875e-13fb5df1476b None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:41.598637 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a104b0-2437-4662-875e-13fb5df1476b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:41.619111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-56a104b0-2437-4662-875e-13fb5df1476b None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:02:41.620435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-56a104b0-2437-4662-875e-13fb5df1476b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:02:41.629238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 958/1903] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:02:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:02:41.634258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6217ac46-1e79-49fe-8325-ae7e16bf3404 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:02:41.645777 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6217ac46-1e79-49fe-8325-ae7e16bf3404 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:02:41.652114 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c68488-15ba-4840-a19c-b1f3663db80a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:02:41.652704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c68488-15ba-4840-a19c-b1f3663db80a admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:41.652975 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c68488-15ba-4840-a19c-b1f3663db80a admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:41.653231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c68488-15ba-4840-a19c-b1f3663db80a admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:41.653870 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c68488-15ba-4840-a19c-b1f3663db80a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:02:41.654163 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c68488-15ba-4840-a19c-b1f3663db80a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:02:41.655748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c68488-15ba-4840-a19c-b1f3663db80a admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:02:41.667967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 946/1904] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:02:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 01:02:41.674830 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a104b0-2437-4662-875e-13fb5df1476b None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:02:41.689689 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a104b0-2437-4662-875e-13fb5df1476b None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:02:41.699170 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fa47a8-c97c-44ca-98b8-5665ee978def admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:02:41.699770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fa47a8-c97c-44ca-98b8-5665ee978def admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:41.700071 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fa47a8-c97c-44ca-98b8-5665ee978def admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:41.700335 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fa47a8-c97c-44ca-98b8-5665ee978def admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:41.700955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fa47a8-c97c-44ca-98b8-5665ee978def admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:02:41.702857 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fa47a8-c97c-44ca-98b8-5665ee978def admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:02:41.737529 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 959/1905] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:02:41 2022] POST /identity/v3/projects => generated 385 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:02:41.742503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c68488-15ba-4840-a19c-b1f3663db80a admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:02:41.754411 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c68488-15ba-4840-a19c-b1f3663db80a admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:02:41.758306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:02:41.815201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa626a61-dca4-4312-a883-23f4dfe227b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:02:41.815871 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa626a61-dca4-4312-a883-23f4dfe227b8 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:41.816148 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa626a61-dca4-4312-a883-23f4dfe227b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:41.816416 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa626a61-dca4-4312-a883-23f4dfe227b8 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:41.817059 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa626a61-dca4-4312-a883-23f4dfe227b8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:02:41.819026 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa626a61-dca4-4312-a883-23f4dfe227b8 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:02:41.843252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 947/1906] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:02:41 2022] POST /identity/v3/users => generated 293 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:02:41.847738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fa47a8-c97c-44ca-98b8-5665ee978def admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:02:41.858255 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fa47a8-c97c-44ca-98b8-5665ee978def admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:02:41.863192 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dece64-bb88-4933-83e3-ce70900522b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:02:41.863857 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dece64-bb88-4933-83e3-ce70900522b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:41.864121 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dece64-bb88-4933-83e3-ce70900522b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:41.864383 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dece64-bb88-4933-83e3-ce70900522b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:41.865010 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34dece64-bb88-4933-83e3-ce70900522b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:02:41.866588 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34dece64-bb88-4933-83e3-ce70900522b2 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:02:41.876975 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 960/1907] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:02:41 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:02:41.882526 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa626a61-dca4-4312-a883-23f4dfe227b8 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:02:41.893423 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa626a61-dca4-4312-a883-23f4dfe227b8 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:02:41.898667 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f617d807-1a12-47b1-a5aa-a9d2651e60d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:02:41.899280 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f617d807-1a12-47b1-a5aa-a9d2651e60d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:41.899553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f617d807-1a12-47b1-a5aa-a9d2651e60d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:41.899890 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f617d807-1a12-47b1-a5aa-a9d2651e60d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:41.900519 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f617d807-1a12-47b1-a5aa-a9d2651e60d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:02:41.902202 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f617d807-1a12-47b1-a5aa-a9d2651e60d8 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:02:41.912995 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 948/1908] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:02:41 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:02:41.917135 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dece64-bb88-4933-83e3-ce70900522b2 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:02:41.927225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dece64-bb88-4933-83e3-ce70900522b2 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:02:41.932520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d1f8d4-ed42-4baf-b2b1-749a3af93090 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:02:41.933232 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d1f8d4-ed42-4baf-b2b1-749a3af93090 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:41.933662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d1f8d4-ed42-4baf-b2b1-749a3af93090 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:41.933934 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d1f8d4-ed42-4baf-b2b1-749a3af93090 admin admin] PATH_INFO: `/v3/projects/1dcbf2e9fb2e4a4698633fcbb07c30cd/users/05eed70effab4e38ae469ab44d1114db/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:41.952839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d1f8d4-ed42-4baf-b2b1-749a3af93090 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1dcbf2e9fb2e4a4698633fcbb07c30cd, user_id=05eed70effab4e38ae469ab44d1114db, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:02:41.957391 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d1f8d4-ed42-4baf-b2b1-749a3af93090 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:02:41.974025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 961/1909] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:02:41 2022] PUT /identity/v3/projects/1dcbf2e9fb2e4a4698633fcbb07c30cd/users/05eed70effab4e38ae469ab44d1114db/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:02:41.979956 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934bac4e-fcec-4d75-b680-7329b30c6e30 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:41.980608 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934bac4e-fcec-4d75-b680-7329b30c6e30 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:41.981227 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934bac4e-fcec-4d75-b680-7329b30c6e30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:42.004993 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-934bac4e-fcec-4d75-b680-7329b30c6e30 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:02:42.006208 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-934bac4e-fcec-4d75-b680-7329b30c6e30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:02:42.069384 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 949/1910] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:02:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:02:42.082485 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d1f8d4-ed42-4baf-b2b1-749a3af93090 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:02:42.084467 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-49d1f8d4-ed42-4baf-b2b1-749a3af93090 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:02:42.116816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:02:42.145932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d1f8d4-ed42-4baf-b2b1-749a3af93090 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:02:42.149980 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa92f53f-09c9-41e7-b4ab-d4e6e03d9022 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:02:42.150555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa92f53f-09c9-41e7-b4ab-d4e6e03d9022 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:42.150827 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa92f53f-09c9-41e7-b4ab-d4e6e03d9022 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:42.151091 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa92f53f-09c9-41e7-b4ab-d4e6e03d9022 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:42.160631 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa92f53f-09c9-41e7-b4ab-d4e6e03d9022 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:02:42.162143 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa92f53f-09c9-41e7-b4ab-d4e6e03d9022 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:02:42.172081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 962/1911] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:02:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:02:46.250042 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4d3e56-1e90-4a22-bbc3-ffaa84383408 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:46.250496 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4d3e56-1e90-4a22-bbc3-ffaa84383408 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:46.250496 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4d3e56-1e90-4a22-bbc3-ffaa84383408 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:46.286105 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-5d4d3e56-1e90-4a22-bbc3-ffaa84383408 None None] MFA Rules not processed for user `05eed70effab4e38ae469ab44d1114db`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:02:46.288577 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-5d4d3e56-1e90-4a22-bbc3-ffaa84383408 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:02:46.387352 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 950/1912] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:02:46 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:02:46.392302 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa92f53f-09c9-41e7-b4ab-d4e6e03d9022 service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:02:46.402509 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa92f53f-09c9-41e7-b4ab-d4e6e03d9022 service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:02:46.407045 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21f7ddd-be27-46ed-a762-0b774ff182a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:02:46.407654 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21f7ddd-be27-46ed-a762-0b774ff182a5 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:46.407951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21f7ddd-be27-46ed-a762-0b774ff182a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:46.408227 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21f7ddd-be27-46ed-a762-0b774ff182a5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:46.408866 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21f7ddd-be27-46ed-a762-0b774ff182a5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:02:46.410772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21f7ddd-be27-46ed-a762-0b774ff182a5 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:02:46.443904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 963/1913] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:02:46 2022] POST /identity/v3/projects => generated 385 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:02:46.445178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4d3e56-1e90-4a22-bbc3-ffaa84383408 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:02:46.457426 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4d3e56-1e90-4a22-bbc3-ffaa84383408 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:02:46.469561 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:02:46.518132 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2695d3-6e1c-479e-9e18-03e1263b3809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:02:46.518752 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2695d3-6e1c-479e-9e18-03e1263b3809 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:46.519027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2695d3-6e1c-479e-9e18-03e1263b3809 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:46.519461 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2695d3-6e1c-479e-9e18-03e1263b3809 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:46.520137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2695d3-6e1c-479e-9e18-03e1263b3809 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:02:46.522342 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2695d3-6e1c-479e-9e18-03e1263b3809 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:02:46.546528 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 951/1914] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:02:46 2022] POST /identity/v3/users => generated 299 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:02:46.550769 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21f7ddd-be27-46ed-a762-0b774ff182a5 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:02:46.561565 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21f7ddd-be27-46ed-a762-0b774ff182a5 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:02:46.566075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e634df7-c091-4adc-8fd1-ac48ef323370 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:02:46.566692 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e634df7-c091-4adc-8fd1-ac48ef323370 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:46.566965 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e634df7-c091-4adc-8fd1-ac48ef323370 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:46.567234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e634df7-c091-4adc-8fd1-ac48ef323370 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:46.568097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e634df7-c091-4adc-8fd1-ac48ef323370 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:02:46.569554 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e634df7-c091-4adc-8fd1-ac48ef323370 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:02:46.580856 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 964/1915] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:02:46 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:02:46.585497 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2695d3-6e1c-479e-9e18-03e1263b3809 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:02:46.596004 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2695d3-6e1c-479e-9e18-03e1263b3809 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:02:46.600957 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd7be37-14a2-4050-a580-58fc91e004e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:02:46.601633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd7be37-14a2-4050-a580-58fc91e004e8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:46.601911 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd7be37-14a2-4050-a580-58fc91e004e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:46.602182 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd7be37-14a2-4050-a580-58fc91e004e8 admin admin] PATH_INFO: `/v3/projects/a2e086e391bc4489bba3032b906ca722/users/7e83347263024b759e9a65e971336335/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:46.617979 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd7be37-14a2-4050-a580-58fc91e004e8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a2e086e391bc4489bba3032b906ca722, user_id=7e83347263024b759e9a65e971336335, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:02:46.620753 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd7be37-14a2-4050-a580-58fc91e004e8 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:02:46.636372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 952/1916] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:02:46 2022] PUT /identity/v3/projects/a2e086e391bc4489bba3032b906ca722/users/7e83347263024b759e9a65e971336335/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:02:46.640597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e634df7-c091-4adc-8fd1-ac48ef323370 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:02:46.651518 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e634df7-c091-4adc-8fd1-ac48ef323370 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:02:46.730480 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46758509-ad08-4582-bfe1-15a3ebabd1af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:02:46.731100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46758509-ad08-4582-bfe1-15a3ebabd1af admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:46.731404 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46758509-ad08-4582-bfe1-15a3ebabd1af admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:46.731668 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46758509-ad08-4582-bfe1-15a3ebabd1af admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:46.732339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46758509-ad08-4582-bfe1-15a3ebabd1af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:02:46.733941 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46758509-ad08-4582-bfe1-15a3ebabd1af admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:02:46.743802 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 965/1917] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:02:46 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:02:46.748166 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd7be37-14a2-4050-a580-58fc91e004e8 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:02:46.761116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd7be37-14a2-4050-a580-58fc91e004e8 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:02:46.766131 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff195e1a-b447-4bb6-8d3a-92585ef6084f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:02:46.766895 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff195e1a-b447-4bb6-8d3a-92585ef6084f admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:46.767266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff195e1a-b447-4bb6-8d3a-92585ef6084f admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:46.767604 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff195e1a-b447-4bb6-8d3a-92585ef6084f admin admin] PATH_INFO: `/v3/projects/a2e086e391bc4489bba3032b906ca722/users/7e83347263024b759e9a65e971336335/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:46.769631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff195e1a-b447-4bb6-8d3a-92585ef6084f admin admin] RBAC: Authorizing `identity:create_grant(project_id=a2e086e391bc4489bba3032b906ca722, user_id=7e83347263024b759e9a65e971336335, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:02:46.772505 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff195e1a-b447-4bb6-8d3a-92585ef6084f admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:02:46.783853 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 953/1918] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:02:46 2022] PUT /identity/v3/projects/a2e086e391bc4489bba3032b906ca722/users/7e83347263024b759e9a65e971336335/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:02:51.042493 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c6a31c-4ccd-4b70-9334-cef5e2f8da75 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:51.043132 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c6a31c-4ccd-4b70-9334-cef5e2f8da75 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:51.043132 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c6a31c-4ccd-4b70-9334-cef5e2f8da75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:51.072575 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-80c6a31c-4ccd-4b70-9334-cef5e2f8da75 None None] MFA Rules not processed for user `7e83347263024b759e9a65e971336335`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:02:51.073742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-80c6a31c-4ccd-4b70-9334-cef5e2f8da75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:02:51.144908 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 966/1919] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:02:51 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:02:51.157563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff195e1a-b447-4bb6-8d3a-92585ef6084f admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:02:51.159424 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-ff195e1a-b447-4bb6-8d3a-92585ef6084f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:02:51.222997 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff195e1a-b447-4bb6-8d3a-92585ef6084f admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:02:51.227318 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302ef875-fbb4-4b3d-be5b-cabc1c79ab8f service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '807b123652d245269336fcffaf36478e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:02:51.227960 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302ef875-fbb4-4b3d-be5b-cabc1c79ab8f service glance] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:51.228234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302ef875-fbb4-4b3d-be5b-cabc1c79ab8f service glance] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:51.228498 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302ef875-fbb4-4b3d-be5b-cabc1c79ab8f service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:51.291125 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-302ef875-fbb4-4b3d-be5b-cabc1c79ab8f service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:02:51.293267 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-302ef875-fbb4-4b3d-be5b-cabc1c79ab8f service glance] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:02:51.302928 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 954/1920] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:02:51 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:02:51.448509 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8e4a14-6262-4343-9865-8c914fb589dd None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:51.448967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8e4a14-6262-4343-9865-8c914fb589dd None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:51.449290 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8e4a14-6262-4343-9865-8c914fb589dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:51.468792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-ba8e4a14-6262-4343-9865-8c914fb589dd None None] MFA Rules not processed for user `4716e7afe2784e2a9b872996d895c374`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:02:51.469990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ba8e4a14-6262-4343-9865-8c914fb589dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:02:51.484597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:02:51.532589 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 967/1921] 10.222.0.16 () {58 vars in 1009 bytes} [Sun Oct 30 01:02:51 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:02:51.545613 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302ef875-fbb4-4b3d-be5b-cabc1c79ab8f service glance] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:02:51.547559 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-302ef875-fbb4-4b3d-be5b-cabc1c79ab8f service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:02:51.575665 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:02:51.622257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302ef875-fbb4-4b3d-be5b-cabc1c79ab8f service glance] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:02:51.627273 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fd5d70-62d6-4bb9-bcf0-e0ffa1503d2e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '67f6eff8c0254cc6a210101dd7af988d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:02:51.627946 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fd5d70-62d6-4bb9-bcf0-e0ffa1503d2e service swift] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:02:51.628217 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fd5d70-62d6-4bb9-bcf0-e0ffa1503d2e service swift] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:02:51.628486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fd5d70-62d6-4bb9-bcf0-e0ffa1503d2e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:02:51.637602 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fd5d70-62d6-4bb9-bcf0-e0ffa1503d2e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:02:51.639199 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fd5d70-62d6-4bb9-bcf0-e0ffa1503d2e service swift] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:02:51.649847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 955/1922] 10.222.0.16 () {60 vars in 1384 bytes} [Sun Oct 30 01:02:51 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 30 01:03:00.544847 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4839eaa-8162-43ac-b780-e79af91a3f68 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:03:00.545351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4839eaa-8162-43ac-b780-e79af91a3f68 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:03:00.545619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4839eaa-8162-43ac-b780-e79af91a3f68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:03:00.570098 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-f4839eaa-8162-43ac-b780-e79af91a3f68 None None] MFA Rules not processed for user `05eed70effab4e38ae469ab44d1114db`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:03:00.571282 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-f4839eaa-8162-43ac-b780-e79af91a3f68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:03:00.579274 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 968/1923] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:03:00 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:03:01.644412 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=22 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:03:03.037493 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4054cff-8ad0-436f-9053-db8446c8a6e7 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:03:03.038072 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4054cff-8ad0-436f-9053-db8446c8a6e7 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:03:03.038072 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4054cff-8ad0-436f-9053-db8446c8a6e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:03:03.058042 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-c4054cff-8ad0-436f-9053-db8446c8a6e7 None None] MFA Rules not processed for user `4716e7afe2784e2a9b872996d895c374`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:03:03.059321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-c4054cff-8ad0-436f-9053-db8446c8a6e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:03:03.067179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 956/1924] 10.222.0.16 () {58 vars in 1009 bytes} [Sun Oct 30 01:03:03 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:03:03.079747 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4839eaa-8162-43ac-b780-e79af91a3f68 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:03:03.092305 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4839eaa-8162-43ac-b780-e79af91a3f68 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:03:03.098451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b3d7a5-32c6-474c-abc3-1b0c9bcf6d09 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '67f6eff8c0254cc6a210101dd7af988d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:03:03.099057 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b3d7a5-32c6-474c-abc3-1b0c9bcf6d09 service swift] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:03:03.099325 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b3d7a5-32c6-474c-abc3-1b0c9bcf6d09 service swift] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:03:03.099602 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b3d7a5-32c6-474c-abc3-1b0c9bcf6d09 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:03:03.108752 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b3d7a5-32c6-474c-abc3-1b0c9bcf6d09 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:03:03.110320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b3d7a5-32c6-474c-abc3-1b0c9bcf6d09 service swift] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:03:03.118837 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 969/1925] 10.222.0.16 () {60 vars in 1384 bytes} [Sun Oct 30 01:03:03 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 30 01:03:13.052291 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:03:13.115619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=26 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:03:39.117640 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4054cff-8ad0-436f-9053-db8446c8a6e7 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:03:39.119265 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-c4054cff-8ad0-436f-9053-db8446c8a6e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:03:39.177723 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4054cff-8ad0-436f-9053-db8446c8a6e7 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:03:39.181821 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0dbbfd-aeb6-4e6d-9ee3-a7ae32547a4f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:03:39.182472 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0dbbfd-aeb6-4e6d-9ee3-a7ae32547a4f service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:03:39.182742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0dbbfd-aeb6-4e6d-9ee3-a7ae32547a4f service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:03:39.183011 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0dbbfd-aeb6-4e6d-9ee3-a7ae32547a4f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:03:39.192849 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0dbbfd-aeb6-4e6d-9ee3-a7ae32547a4f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:03:39.194298 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0dbbfd-aeb6-4e6d-9ee3-a7ae32547a4f service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:03:39.204350 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 957/1926] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:03:39 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:03:46.327152 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b3d7a5-32c6-474c-abc3-1b0c9bcf6d09 service swift] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:03:46.328831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-79b3d7a5-32c6-474c-abc3-1b0c9bcf6d09 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:03:46.405527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b3d7a5-32c6-474c-abc3-1b0c9bcf6d09 service swift] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:03:46.412645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b918db4f-a950-4e7f-9271-e6e23d69fdae service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:03:46.413280 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b918db4f-a950-4e7f-9271-e6e23d69fdae service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:03:46.413552 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b918db4f-a950-4e7f-9271-e6e23d69fdae service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:03:46.413825 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b918db4f-a950-4e7f-9271-e6e23d69fdae service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:03:46.415831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-b918db4f-a950-4e7f-9271-e6e23d69fdae service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:03:46.476066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b918db4f-a950-4e7f-9271-e6e23d69fdae service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:03:46.477647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b918db4f-a950-4e7f-9271-e6e23d69fdae service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:03:46.487130 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 970/1927] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:03:46 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:03:49.200207 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=26 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:03:53.771699 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9626aef6-0288-45fa-95a2-0e448b86094e None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:03:53.772316 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9626aef6-0288-45fa-95a2-0e448b86094e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:03:53.772316 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9626aef6-0288-45fa-95a2-0e448b86094e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:03:53.801873 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-9626aef6-0288-45fa-95a2-0e448b86094e None None] MFA Rules not processed for user `ecaf8ec105c04b919256f852da859eba`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:03:53.803063 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-9626aef6-0288-45fa-95a2-0e448b86094e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:03:53.879955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 958/1928] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:03:53 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:03:56.482736 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=56 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:04:03.878199 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=37 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:04:07.669313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8371b75-7ef1-42a1-9925-6a36dd8ad29b None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:04:07.669795 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8371b75-7ef1-42a1-9925-6a36dd8ad29b None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:04:07.669891 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8371b75-7ef1-42a1-9925-6a36dd8ad29b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:04:07.689894 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-c8371b75-7ef1-42a1-9925-6a36dd8ad29b None None] MFA Rules not processed for user `15e603f9898a430d8a53f21818e97722`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:04:07.691076 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c8371b75-7ef1-42a1-9925-6a36dd8ad29b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:04:07.752142 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 971/1929] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:04:07 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:04:17.745021 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=25 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:04:20.425214 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ef516b-70ab-4f8c-b0aa-8d6a872fca18 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:04:20.425678 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ef516b-70ab-4f8c-b0aa-8d6a872fca18 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:04:20.425678 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ef516b-70ab-4f8c-b0aa-8d6a872fca18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:04:20.446770 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-02ef516b-70ab-4f8c-b0aa-8d6a872fca18 None None] MFA Rules not processed for user `31b7342ecf07408cab6bb040f361fa1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:04:20.448037 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-02ef516b-70ab-4f8c-b0aa-8d6a872fca18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:04:20.456253 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 959/1930] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:04:20 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:04:30.441397 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:04:47.851050 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8371b75-7ef1-42a1-9925-6a36dd8ad29b None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:04:47.852697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c8371b75-7ef1-42a1-9925-6a36dd8ad29b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:04:47.972212 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8371b75-7ef1-42a1-9925-6a36dd8ad29b None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:04:47.978447 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b3de19-c1aa-4d83-b11c-998e7ef385f1 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '63cc035057064805a95e9f12714ebade', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:04:47.979066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b3de19-c1aa-4d83-b11c-998e7ef385f1 service placement] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:04:47.979333 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b3de19-c1aa-4d83-b11c-998e7ef385f1 service placement] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:04:47.979604 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b3de19-c1aa-4d83-b11c-998e7ef385f1 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:04:47.981486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-92b3de19-c1aa-4d83-b11c-998e7ef385f1 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:04:47.993269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b3de19-c1aa-4d83-b11c-998e7ef385f1 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:04:47.994848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b3de19-c1aa-4d83-b11c-998e7ef385f1 service placement] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:04:48.005501 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 972/1931] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:04:47 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 01:04:51.670015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3339be74-4d99-46d4-b9f6-e700ce3c0d91 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:04:51.670565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3339be74-4d99-46d4-b9f6-e700ce3c0d91 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:04:51.670565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3339be74-4d99-46d4-b9f6-e700ce3c0d91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:04:51.690204 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-3339be74-4d99-46d4-b9f6-e700ce3c0d91 None None] MFA Rules not processed for user `1c0e5bd11ac14259ad376ce54bcb7025`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:04:51.691474 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-3339be74-4d99-46d4-b9f6-e700ce3c0d91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:04:51.750654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 960/1932] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:04:51 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:04:58.006072 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=51 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:05:01.743286 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=25 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:05:47.455636 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b3de19-c1aa-4d83-b11c-998e7ef385f1 service placement] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:05:47.478525 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b3de19-c1aa-4d83-b11c-998e7ef385f1 service placement] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:05:47.482785 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fbd124-507e-45fb-8c86-eb257863ed59 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:05:47.483678 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fbd124-507e-45fb-8c86-eb257863ed59 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:05:47.484149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fbd124-507e-45fb-8c86-eb257863ed59 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:05:47.484422 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fbd124-507e-45fb-8c86-eb257863ed59 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:05:47.486425 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-d8fbd124-507e-45fb-8c86-eb257863ed59 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:05:47.525311 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fbd124-507e-45fb-8c86-eb257863ed59 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:05:47.526773 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fbd124-507e-45fb-8c86-eb257863ed59 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:05:47.551573 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 973/1933] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:05:47 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:05:52.429714 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3339be74-4d99-46d4-b9f6-e700ce3c0d91 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:05:52.443610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3339be74-4d99-46d4-b9f6-e700ce3c0d91 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:05:52.447983 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0dfe55-48a8-47d6-8e74-a62e6f16a8c0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:05:52.448594 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0dfe55-48a8-47d6-8e74-a62e6f16a8c0 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:05:52.448873 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0dfe55-48a8-47d6-8e74-a62e6f16a8c0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:05:52.449147 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0dfe55-48a8-47d6-8e74-a62e6f16a8c0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:05:52.450929 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-6e0dfe55-48a8-47d6-8e74-a62e6f16a8c0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:05:52.464533 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0dfe55-48a8-47d6-8e74-a62e6f16a8c0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:05:52.465959 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0dfe55-48a8-47d6-8e74-a62e6f16a8c0 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:05:52.477409 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 961/1934] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:05:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:05:54.715409 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fbd124-507e-45fb-8c86-eb257863ed59 service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:05:54.724843 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fbd124-507e-45fb-8c86-eb257863ed59 service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:05:54.728493 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c12c50-289b-40b6-a250-692ef6e87e5b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:05:54.729101 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c12c50-289b-40b6-a250-692ef6e87e5b service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:05:54.729372 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c12c50-289b-40b6-a250-692ef6e87e5b service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:05:54.729647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c12c50-289b-40b6-a250-692ef6e87e5b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:05:54.731408 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-42c12c50-289b-40b6-a250-692ef6e87e5b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:05:54.816406 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c12c50-289b-40b6-a250-692ef6e87e5b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:05:54.817882 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c12c50-289b-40b6-a250-692ef6e87e5b service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:05:54.837401 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 974/1935] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:05:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:05:58.648912 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0dfe55-48a8-47d6-8e74-a62e6f16a8c0 service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:05:58.661754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0dfe55-48a8-47d6-8e74-a62e6f16a8c0 service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:05:58.667508 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6066d7d0-5d4e-41f1-b81a-4d8de640d2db service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '63cc035057064805a95e9f12714ebade', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:05:58.668168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6066d7d0-5d4e-41f1-b81a-4d8de640d2db service placement] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:05:58.668447 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6066d7d0-5d4e-41f1-b81a-4d8de640d2db service placement] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:05:58.668906 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6066d7d0-5d4e-41f1-b81a-4d8de640d2db service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:05:58.670838 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-6066d7d0-5d4e-41f1-b81a-4d8de640d2db service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:05:58.684602 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6066d7d0-5d4e-41f1-b81a-4d8de640d2db service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:05:58.686292 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6066d7d0-5d4e-41f1-b81a-4d8de640d2db service placement] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:05:58.713819 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 962/1936] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:05:58 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:05:58.725316 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c12c50-289b-40b6-a250-692ef6e87e5b service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:05:58.727694 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-42c12c50-289b-40b6-a250-692ef6e87e5b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:05:58.742757 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c12c50-289b-40b6-a250-692ef6e87e5b service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:05:58.750154 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f76c5f6-f5d8-4628-b67e-0ce4ef923f62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:05:58.751287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f76c5f6-f5d8-4628-b67e-0ce4ef923f62 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:05:58.751790 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f76c5f6-f5d8-4628-b67e-0ce4ef923f62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:05:58.752147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f76c5f6-f5d8-4628-b67e-0ce4ef923f62 admin admin] PATH_INFO: `/v3/users/31b7342ecf07408cab6bb040f361fa1f` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:05:58.753480 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f76c5f6-f5d8-4628-b67e-0ce4ef923f62 admin admin] RBAC: Authorizing `identity:delete_user(user_id=31b7342ecf07408cab6bb040f361fa1f)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:05:58.755608 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f76c5f6-f5d8-4628-b67e-0ce4ef923f62 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:05:58.807044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8f76c5f6-f5d8-4628-b67e-0ce4ef923f62 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '31b7342ecf07408cab6bb040f361fa1f'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:05:58.815941 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8f76c5f6-f5d8-4628-b67e-0ce4ef923f62 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '31b7342ecf07408cab6bb040f361fa1f'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:05:58.816556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8f76c5f6-f5d8-4628-b67e-0ce4ef923f62 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '31b7342ecf07408cab6bb040f361fa1f'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:05:58.829421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8f76c5f6-f5d8-4628-b67e-0ce4ef923f62 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '31b7342ecf07408cab6bb040f361fa1f'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:05:58.830567 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:05:58.837377 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8f76c5f6-f5d8-4628-b67e-0ce4ef923f62 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '31b7342ecf07408cab6bb040f361fa1f'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:05:58.844654 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 975/1937] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:05:58 2022] DELETE /identity/v3/users/31b7342ecf07408cab6bb040f361fa1f => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:05:59.023445 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6066d7d0-5d4e-41f1-b81a-4d8de640d2db service placement] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:05:59.042473 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6066d7d0-5d4e-41f1-b81a-4d8de640d2db service placement] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:05:59.103012 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c48eca9-9cee-495c-9f34-4381cdd77e0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:05:59.103836 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c48eca9-9cee-495c-9f34-4381cdd77e0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:05:59.104190 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c48eca9-9cee-495c-9f34-4381cdd77e0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:05:59.104482 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c48eca9-9cee-495c-9f34-4381cdd77e0b admin admin] PATH_INFO: `/v3/projects/f48276ac51dd4bb1912fc670433cf6c4` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:05:59.105755 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c48eca9-9cee-495c-9f34-4381cdd77e0b admin admin] RBAC: Authorizing `identity:delete_project(project_id=f48276ac51dd4bb1912fc670433cf6c4)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:05:59.107953 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c48eca9-9cee-495c-9f34-4381cdd77e0b admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:05:59.160588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4c48eca9-9cee-495c-9f34-4381cdd77e0b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f48276ac51dd4bb1912fc670433cf6c4', 'request_id': 'req-4c48eca9-9cee-495c-9f34-4381cdd77e0b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:05:59.167065 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4c48eca9-9cee-495c-9f34-4381cdd77e0b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f48276ac51dd4bb1912fc670433cf6c4', 'request_id': 'req-4c48eca9-9cee-495c-9f34-4381cdd77e0b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:05:59.176144 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4c48eca9-9cee-495c-9f34-4381cdd77e0b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f48276ac51dd4bb1912fc670433cf6c4', 'request_id': 'req-4c48eca9-9cee-495c-9f34-4381cdd77e0b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:05:59.189685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4c48eca9-9cee-495c-9f34-4381cdd77e0b admin admin] The token cache is being invalidate because project f48276ac51dd4bb1912fc670433cf6c4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:05:59.189685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4c48eca9-9cee-495c-9f34-4381cdd77e0b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:05:59.190337 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-4c48eca9-9cee-495c-9f34-4381cdd77e0b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:05:59.195467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 963/1938] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:05:59 2022] DELETE /identity/v3/projects/f48276ac51dd4bb1912fc670433cf6c4 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:05:59.197311 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902cd3c3-5971-44d2-848d-3a09ab72f721 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:05:59.197598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902cd3c3-5971-44d2-848d-3a09ab72f721 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:05:59.197869 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902cd3c3-5971-44d2-848d-3a09ab72f721 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:05:59.259233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-902cd3c3-5971-44d2-848d-3a09ab72f721 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:05:59.260476 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-902cd3c3-5971-44d2-848d-3a09ab72f721 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:05:59.318872 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 976/1939] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:05:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:05:59.326897 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447d0d83-d8cb-4960-a26c-139428d192fb None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:05:59.327187 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447d0d83-d8cb-4960-a26c-139428d192fb None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:05:59.327457 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447d0d83-d8cb-4960-a26c-139428d192fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:05:59.348693 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-447d0d83-d8cb-4960-a26c-139428d192fb None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:05:59.350013 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-447d0d83-d8cb-4960-a26c-139428d192fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:05:59.357874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 964/1940] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:05:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:05:59.362426 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902cd3c3-5971-44d2-848d-3a09ab72f721 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:05:59.373977 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902cd3c3-5971-44d2-848d-3a09ab72f721 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:05:59.379025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c26096-4794-45ec-b372-1481ae384e75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:05:59.380239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c26096-4794-45ec-b372-1481ae384e75 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:05:59.380239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c26096-4794-45ec-b372-1481ae384e75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:05:59.380487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c26096-4794-45ec-b372-1481ae384e75 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:05:59.381140 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c26096-4794-45ec-b372-1481ae384e75 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:05:59.381437 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c26096-4794-45ec-b372-1481ae384e75 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:05:59.383010 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c26096-4794-45ec-b372-1481ae384e75 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:05:59.395155 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 977/1941] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:05:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 01:05:59.399632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447d0d83-d8cb-4960-a26c-139428d192fb None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:05:59.408959 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447d0d83-d8cb-4960-a26c-139428d192fb None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:05:59.413700 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2b7b80-5b75-44aa-bc0c-66f5c3eecc4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:05:59.414308 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2b7b80-5b75-44aa-bc0c-66f5c3eecc4a admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:05:59.414580 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2b7b80-5b75-44aa-bc0c-66f5c3eecc4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:05:59.415264 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2b7b80-5b75-44aa-bc0c-66f5c3eecc4a admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:05:59.415472 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2b7b80-5b75-44aa-bc0c-66f5c3eecc4a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:05:59.417587 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2b7b80-5b75-44aa-bc0c-66f5c3eecc4a admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:05:59.467114 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 965/1942] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:05:59 2022] POST /identity/v3/projects => generated 405 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:05:59.471374 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c26096-4794-45ec-b372-1481ae384e75 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:05:59.482525 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c26096-4794-45ec-b372-1481ae384e75 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:05:59.561848 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c48b78b-e436-4b9b-b122-c6e94320e4ef None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:05:59.562131 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c48b78b-e436-4b9b-b122-c6e94320e4ef None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:05:59.562397 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c48b78b-e436-4b9b-b122-c6e94320e4ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:05:59.565477 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b81410-729d-4cee-be5b-4e0e6b729cb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:05:59.566190 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b81410-729d-4cee-be5b-4e0e6b729cb0 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:05:59.567506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b81410-729d-4cee-be5b-4e0e6b729cb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:05:59.567900 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b81410-729d-4cee-be5b-4e0e6b729cb0 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:05:59.568625 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b81410-729d-4cee-be5b-4e0e6b729cb0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:05:59.570646 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b81410-729d-4cee-be5b-4e0e6b729cb0 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:05:59.595689 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 978/1943] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:05:59 2022] POST /identity/v3/users => generated 303 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:05:59.600015 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b81410-729d-4cee-be5b-4e0e6b729cb0 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:05:59.603675 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:05:59.614404 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b81410-729d-4cee-be5b-4e0e6b729cb0 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:05:59.619130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-4c48b78b-e436-4b9b-b122-c6e94320e4ef None None] MFA Rules not processed for user `ecaf8ec105c04b919256f852da859eba`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:05:59.619872 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e10c46-17a0-479b-b718-0e8b44abefb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:05:59.620414 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-4c48b78b-e436-4b9b-b122-c6e94320e4ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:05:59.620519 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e10c46-17a0-479b-b718-0e8b44abefb2 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:05:59.620752 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e10c46-17a0-479b-b718-0e8b44abefb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:05:59.621020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e10c46-17a0-479b-b718-0e8b44abefb2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:05:59.621665 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e10c46-17a0-479b-b718-0e8b44abefb2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:05:59.623272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e10c46-17a0-479b-b718-0e8b44abefb2 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:05:59.634925 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 979/1944] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:05:59 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:05:59.638958 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e10c46-17a0-479b-b718-0e8b44abefb2 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:05:59.649063 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e10c46-17a0-479b-b718-0e8b44abefb2 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:05:59.653876 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2440c69-3953-4d34-948b-ee7bd5ac2a8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:05:59.654468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2440c69-3953-4d34-948b-ee7bd5ac2a8f admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:05:59.654738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2440c69-3953-4d34-948b-ee7bd5ac2a8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:05:59.655003 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2440c69-3953-4d34-948b-ee7bd5ac2a8f admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:05:59.655629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2440c69-3953-4d34-948b-ee7bd5ac2a8f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:05:59.657234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2440c69-3953-4d34-948b-ee7bd5ac2a8f admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:05:59.667748 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 980/1945] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:05:59 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:05:59.671605 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2440c69-3953-4d34-948b-ee7bd5ac2a8f admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:05:59.681866 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2440c69-3953-4d34-948b-ee7bd5ac2a8f admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:05:59.686426 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dfadf0-9fe7-40ac-98b2-864593b5efe6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:05:59.687061 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dfadf0-9fe7-40ac-98b2-864593b5efe6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:05:59.687329 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dfadf0-9fe7-40ac-98b2-864593b5efe6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:05:59.687592 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dfadf0-9fe7-40ac-98b2-864593b5efe6 admin admin] PATH_INFO: `/v3/projects/61896e9e6d6040709cd1ceaac8d2942c/users/75f4143107f34ca2ae425ff876d43c57/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:05:59.688245 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 966/1946] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:05:59 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:05:59.702469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:05:59.706660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9dfadf0-9fe7-40ac-98b2-864593b5efe6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=61896e9e6d6040709cd1ceaac8d2942c, user_id=75f4143107f34ca2ae425ff876d43c57, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:05:59.710253 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9dfadf0-9fe7-40ac-98b2-864593b5efe6 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:05:59.726339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 981/1947] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:05:59 2022] PUT /identity/v3/projects/61896e9e6d6040709cd1ceaac8d2942c/users/75f4143107f34ca2ae425ff876d43c57/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:05:59.737844 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81258f2d-fe67-4ea0-82ce-f15e026a535f None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:05:59.738124 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81258f2d-fe67-4ea0-82ce-f15e026a535f None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:05:59.738388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81258f2d-fe67-4ea0-82ce-f15e026a535f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:05:59.767638 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-81258f2d-fe67-4ea0-82ce-f15e026a535f None None] MFA Rules not processed for user `75f4143107f34ca2ae425ff876d43c57`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:05:59.768893 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-81258f2d-fe67-4ea0-82ce-f15e026a535f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:05:59.834803 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 967/1948] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:05:59 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:05:59.839808 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dfadf0-9fe7-40ac-98b2-864593b5efe6 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:05:59.907551 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dfadf0-9fe7-40ac-98b2-864593b5efe6 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:05:59.912696 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ec206c-3ea5-4c6d-84c7-cde2043416f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:05:59.913478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ec206c-3ea5-4c6d-84c7-cde2043416f0 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:05:59.913736 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ec206c-3ea5-4c6d-84c7-cde2043416f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:05:59.914003 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ec206c-3ea5-4c6d-84c7-cde2043416f0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:05:59.914624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ec206c-3ea5-4c6d-84c7-cde2043416f0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:05:59.916771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ec206c-3ea5-4c6d-84c7-cde2043416f0 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:05:59.949810 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 982/1949] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:05:59 2022] POST /identity/v3/projects => generated 403 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:05:59.950931 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81258f2d-fe67-4ea0-82ce-f15e026a535f None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:05:59.961886 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81258f2d-fe67-4ea0-82ce-f15e026a535f None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:00.010514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ec206c-3ea5-4c6d-84c7-cde2043416f0 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:00.012313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-59ec206c-3ea5-4c6d-84c7-cde2043416f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:00.020174 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6216062a-4456-4a6f-aa5f-58e3e6b2ef48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:00.020958 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6216062a-4456-4a6f-aa5f-58e3e6b2ef48 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:00.021230 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6216062a-4456-4a6f-aa5f-58e3e6b2ef48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:00.021496 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6216062a-4456-4a6f-aa5f-58e3e6b2ef48 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:00.022124 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6216062a-4456-4a6f-aa5f-58e3e6b2ef48 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:00.024097 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6216062a-4456-4a6f-aa5f-58e3e6b2ef48 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:00.047161 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 968/1950] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:05:59 2022] POST /identity/v3/users => generated 308 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:06:00.053455 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6216062a-4456-4a6f-aa5f-58e3e6b2ef48 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:00.066469 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6216062a-4456-4a6f-aa5f-58e3e6b2ef48 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:00.071735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a9bee4-4b2c-413c-a3e6-e0ed8e81037e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:00.072360 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a9bee4-4b2c-413c-a3e6-e0ed8e81037e admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:00.072635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a9bee4-4b2c-413c-a3e6-e0ed8e81037e admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:00.072903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a9bee4-4b2c-413c-a3e6-e0ed8e81037e admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:00.073544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a9bee4-4b2c-413c-a3e6-e0ed8e81037e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:00.075444 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a9bee4-4b2c-413c-a3e6-e0ed8e81037e admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:00.085067 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ec206c-3ea5-4c6d-84c7-cde2043416f0 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:00.089237 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 969/1951] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:06:00 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:06:00.089378 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427db213-163a-4c10-9a00-437d63d4ea72 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:00.089879 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427db213-163a-4c10-9a00-437d63d4ea72 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:00.090152 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427db213-163a-4c10-9a00-437d63d4ea72 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:00.090267 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a9bee4-4b2c-413c-a3e6-e0ed8e81037e admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:00.090413 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427db213-163a-4c10-9a00-437d63d4ea72 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:00.092311 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-427db213-163a-4c10-9a00-437d63d4ea72 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:00.096246 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:00.100601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a9bee4-4b2c-413c-a3e6-e0ed8e81037e admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:00.102378 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427db213-163a-4c10-9a00-437d63d4ea72 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:00.103911 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427db213-163a-4c10-9a00-437d63d4ea72 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:00.106524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512f1986-7c55-4cd2-8466-659061c0b6f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:00.107153 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512f1986-7c55-4cd2-8466-659061c0b6f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:00.107426 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512f1986-7c55-4cd2-8466-659061c0b6f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:00.107951 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512f1986-7c55-4cd2-8466-659061c0b6f1 admin admin] PATH_INFO: `/v3/projects/6feee69ec37946468c1929dd560a3190/users/610d4e043e7147f69e28c1f897da9935/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:00.113085 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 983/1952] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:06:00 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 01:06:00.125456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512f1986-7c55-4cd2-8466-659061c0b6f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6feee69ec37946468c1929dd560a3190, user_id=610d4e043e7147f69e28c1f897da9935, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:00.128388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512f1986-7c55-4cd2-8466-659061c0b6f1 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:00.143407 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 970/1953] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:06:00 2022] PUT /identity/v3/projects/6feee69ec37946468c1929dd560a3190/users/610d4e043e7147f69e28c1f897da9935/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:06:00.147691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427db213-163a-4c10-9a00-437d63d4ea72 service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:00.159473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427db213-163a-4c10-9a00-437d63d4ea72 service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:00.217982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ab97e5-293f-4940-94d0-047c80a3dd78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:00.218587 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ab97e5-293f-4940-94d0-047c80a3dd78 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:00.218876 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ab97e5-293f-4940-94d0-047c80a3dd78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:00.219140 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ab97e5-293f-4940-94d0-047c80a3dd78 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:00.219807 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ab97e5-293f-4940-94d0-047c80a3dd78 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:00.221379 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ab97e5-293f-4940-94d0-047c80a3dd78 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:00.234815 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 984/1954] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:06:00 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:06:00.236046 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512f1986-7c55-4cd2-8466-659061c0b6f1 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:00.246571 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512f1986-7c55-4cd2-8466-659061c0b6f1 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:00.251498 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03639d7-3c89-4b0f-b87b-81951e429b08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:00.252174 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03639d7-3c89-4b0f-b87b-81951e429b08 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:00.252889 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03639d7-3c89-4b0f-b87b-81951e429b08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:00.253158 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03639d7-3c89-4b0f-b87b-81951e429b08 admin admin] PATH_INFO: `/v3/projects/6feee69ec37946468c1929dd560a3190/users/610d4e043e7147f69e28c1f897da9935/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:00.255101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b03639d7-3c89-4b0f-b87b-81951e429b08 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6feee69ec37946468c1929dd560a3190, user_id=610d4e043e7147f69e28c1f897da9935, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:00.258013 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b03639d7-3c89-4b0f-b87b-81951e429b08 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:00.270199 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 971/1955] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:06:00 2022] PUT /identity/v3/projects/6feee69ec37946468c1929dd560a3190/users/610d4e043e7147f69e28c1f897da9935/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:00.280871 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44d757e-df3f-4d26-8cae-9066d2b15efa None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:00.281150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44d757e-df3f-4d26-8cae-9066d2b15efa None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:00.281410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44d757e-df3f-4d26-8cae-9066d2b15efa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:00.288933 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:00.313454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-e44d757e-df3f-4d26-8cae-9066d2b15efa None None] MFA Rules not processed for user `610d4e043e7147f69e28c1f897da9935`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:06:00.314665 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e44d757e-df3f-4d26-8cae-9066d2b15efa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:00.391826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 985/1956] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:06:00 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:06:00.402826 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03639d7-3c89-4b0f-b87b-81951e429b08 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:00.404591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b03639d7-3c89-4b0f-b87b-81951e429b08 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:00.482278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03639d7-3c89-4b0f-b87b-81951e429b08 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:00.486883 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a011161b-2115-4e41-9c89-965d1d750034 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:00.487512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a011161b-2115-4e41-9c89-965d1d750034 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:00.487874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a011161b-2115-4e41-9c89-965d1d750034 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:00.488143 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a011161b-2115-4e41-9c89-965d1d750034 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:00.498516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a011161b-2115-4e41-9c89-965d1d750034 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:00.499981 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a011161b-2115-4e41-9c89-965d1d750034 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:00.511699 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 972/1957] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:06:00 2022] GET /identity/v3/auth/tokens => generated 3347 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:06:00.674152 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44d757e-df3f-4d26-8cae-9066d2b15efa None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:00.740142 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44d757e-df3f-4d26-8cae-9066d2b15efa None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:00.745316 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84633184-6476-46bc-96a2-c2b275dffa61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:00.745959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84633184-6476-46bc-96a2-c2b275dffa61 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:00.746232 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84633184-6476-46bc-96a2-c2b275dffa61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:00.746496 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84633184-6476-46bc-96a2-c2b275dffa61 admin admin] PATH_INFO: `/v3/users/75f4143107f34ca2ae425ff876d43c57` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:00.747814 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84633184-6476-46bc-96a2-c2b275dffa61 admin admin] RBAC: Authorizing `identity:delete_user(user_id=75f4143107f34ca2ae425ff876d43c57)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:00.749742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84633184-6476-46bc-96a2-c2b275dffa61 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:00.798900 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-84633184-6476-46bc-96a2-c2b275dffa61 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '75f4143107f34ca2ae425ff876d43c57'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:00.808279 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-84633184-6476-46bc-96a2-c2b275dffa61 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '75f4143107f34ca2ae425ff876d43c57'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:00.808855 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-84633184-6476-46bc-96a2-c2b275dffa61 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '75f4143107f34ca2ae425ff876d43c57'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:00.821474 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-84633184-6476-46bc-96a2-c2b275dffa61 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '75f4143107f34ca2ae425ff876d43c57'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:00.828383 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-84633184-6476-46bc-96a2-c2b275dffa61 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '75f4143107f34ca2ae425ff876d43c57'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:00.835037 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 986/1958] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:06:00 2022] DELETE /identity/v3/users/75f4143107f34ca2ae425ff876d43c57 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:00.840085 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93dd7f8-a817-4839-bf0d-7fd77e6f359e None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:00.840569 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93dd7f8-a817-4839-bf0d-7fd77e6f359e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:00.840667 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93dd7f8-a817-4839-bf0d-7fd77e6f359e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:00.868731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-e93dd7f8-a817-4839-bf0d-7fd77e6f359e None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:06:00.870054 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-e93dd7f8-a817-4839-bf0d-7fd77e6f359e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:00.929461 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 973/1959] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:06:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:06:00.939746 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84633184-6476-46bc-96a2-c2b275dffa61 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:00.941296 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-84633184-6476-46bc-96a2-c2b275dffa61 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:00.951587 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:01.000656 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84633184-6476-46bc-96a2-c2b275dffa61 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:01.004389 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f561d7-4ee4-4e4d-a1f4-31c9acd76960 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:01.005023 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f561d7-4ee4-4e4d-a1f4-31c9acd76960 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:01.005298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f561d7-4ee4-4e4d-a1f4-31c9acd76960 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:01.005546 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f561d7-4ee4-4e4d-a1f4-31c9acd76960 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:01.014712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f561d7-4ee4-4e4d-a1f4-31c9acd76960 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:01.016174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f561d7-4ee4-4e4d-a1f4-31c9acd76960 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:01.025409 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 987/1960] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:06:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 01:06:01.326290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93dd7f8-a817-4839-bf0d-7fd77e6f359e None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:01.327827 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-e93dd7f8-a817-4839-bf0d-7fd77e6f359e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:01.341572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93dd7f8-a817-4839-bf0d-7fd77e6f359e None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:01.347532 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63f3d6b-269a-4ec6-bbc9-d75e6c846a2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:01.348280 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63f3d6b-269a-4ec6-bbc9-d75e6c846a2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:01.349118 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63f3d6b-269a-4ec6-bbc9-d75e6c846a2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:01.349118 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63f3d6b-269a-4ec6-bbc9-d75e6c846a2c admin admin] PATH_INFO: `/v3/projects/61896e9e6d6040709cd1ceaac8d2942c` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:01.350410 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63f3d6b-269a-4ec6-bbc9-d75e6c846a2c admin admin] RBAC: Authorizing `identity:delete_project(project_id=61896e9e6d6040709cd1ceaac8d2942c)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:01.352598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63f3d6b-269a-4ec6-bbc9-d75e6c846a2c admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:01.415327 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:01.418413 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a63f3d6b-269a-4ec6-bbc9-d75e6c846a2c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '61896e9e6d6040709cd1ceaac8d2942c', 'request_id': 'req-a63f3d6b-269a-4ec6-bbc9-d75e6c846a2c'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:01.426089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f561d7-4ee4-4e4d-a1f4-31c9acd76960 service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:01.427105 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a63f3d6b-269a-4ec6-bbc9-d75e6c846a2c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '61896e9e6d6040709cd1ceaac8d2942c', 'request_id': 'req-a63f3d6b-269a-4ec6-bbc9-d75e6c846a2c'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:01.427994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-01f561d7-4ee4-4e4d-a1f4-31c9acd76960 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:01.438471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a63f3d6b-269a-4ec6-bbc9-d75e6c846a2c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '61896e9e6d6040709cd1ceaac8d2942c', 'request_id': 'req-a63f3d6b-269a-4ec6-bbc9-d75e6c846a2c'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:01.451413 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a63f3d6b-269a-4ec6-bbc9-d75e6c846a2c admin admin] The token cache is being invalidate because project 61896e9e6d6040709cd1ceaac8d2942c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:06:01.451706 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a63f3d6b-269a-4ec6-bbc9-d75e6c846a2c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:01.452377 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a63f3d6b-269a-4ec6-bbc9-d75e6c846a2c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:01.454142 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 974/1961] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:06:01 2022] DELETE /identity/v3/projects/61896e9e6d6040709cd1ceaac8d2942c => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:01.457704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63f3d6b-269a-4ec6-bbc9-d75e6c846a2c admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:01.459218 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a63f3d6b-269a-4ec6-bbc9-d75e6c846a2c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:01.522923 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f561d7-4ee4-4e4d-a1f4-31c9acd76960 service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:01.532004 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63f3d6b-269a-4ec6-bbc9-d75e6c846a2c admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:01.538042 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ba3223-b148-4f69-86d6-2f578f755fac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:01.538687 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ba3223-b148-4f69-86d6-2f578f755fac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:01.538961 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ba3223-b148-4f69-86d6-2f578f755fac admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:01.539228 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ba3223-b148-4f69-86d6-2f578f755fac admin admin] PATH_INFO: `/v3/users/610d4e043e7147f69e28c1f897da9935` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:01.540603 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ba3223-b148-4f69-86d6-2f578f755fac admin admin] RBAC: Authorizing `identity:delete_user(user_id=610d4e043e7147f69e28c1f897da9935)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:01.542729 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ba3223-b148-4f69-86d6-2f578f755fac admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:01.598818 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd077534-02b0-44e8-b2b2-d4fef0e2d034 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:01.599526 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd077534-02b0-44e8-b2b2-d4fef0e2d034 service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:01.599853 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd077534-02b0-44e8-b2b2-d4fef0e2d034 service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:01.600126 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd077534-02b0-44e8-b2b2-d4fef0e2d034 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:01.601815 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-29ba3223-b148-4f69-86d6-2f578f755fac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '610d4e043e7147f69e28c1f897da9935'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:01.602059 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-bd077534-02b0-44e8-b2b2-d4fef0e2d034 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:01.611688 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-29ba3223-b148-4f69-86d6-2f578f755fac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '610d4e043e7147f69e28c1f897da9935'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:01.612643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-29ba3223-b148-4f69-86d6-2f578f755fac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '610d4e043e7147f69e28c1f897da9935'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:01.623522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-29ba3223-b148-4f69-86d6-2f578f755fac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '610d4e043e7147f69e28c1f897da9935'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:01.640104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-29ba3223-b148-4f69-86d6-2f578f755fac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '610d4e043e7147f69e28c1f897da9935'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:01.647558 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 975/1962] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:06:01 2022] DELETE /identity/v3/users/610d4e043e7147f69e28c1f897da9935 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:06:01.656018 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:01.675731 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd077534-02b0-44e8-b2b2-d4fef0e2d034 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:01.677587 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd077534-02b0-44e8-b2b2-d4fef0e2d034 service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:01.687507 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 988/1963] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:06:01 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 262 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 01:06:01.773018 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ba3223-b148-4f69-86d6-2f578f755fac admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:01.775181 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-29ba3223-b148-4f69-86d6-2f578f755fac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:01.851612 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:01.879656 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ba3223-b148-4f69-86d6-2f578f755fac admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:01.885116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ac8140-421d-4be8-aa47-5b3b01219192 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:01.885725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ac8140-421d-4be8-aa47-5b3b01219192 service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:01.885999 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ac8140-421d-4be8-aa47-5b3b01219192 service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:01.886260 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ac8140-421d-4be8-aa47-5b3b01219192 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:01.888028 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d3ac8140-421d-4be8-aa47-5b3b01219192 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:01.972454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd077534-02b0-44e8-b2b2-d4fef0e2d034 service nova] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:01.987787 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd077534-02b0-44e8-b2b2-d4fef0e2d034 service nova] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:01.989435 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ac8140-421d-4be8-aa47-5b3b01219192 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:01.990797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ac8140-421d-4be8-aa47-5b3b01219192 service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:02.014788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 976/1964] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:06:01 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 243 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:06:02.064562 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8161f52-83e9-4a81-9930-8f18d3b388c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:02.065225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8161f52-83e9-4a81-9930-8f18d3b388c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:02.065504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8161f52-83e9-4a81-9930-8f18d3b388c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:02.065771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8161f52-83e9-4a81-9930-8f18d3b388c4 admin admin] PATH_INFO: `/v3/projects/6feee69ec37946468c1929dd560a3190` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:02.066876 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8161f52-83e9-4a81-9930-8f18d3b388c4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6feee69ec37946468c1929dd560a3190)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:02.069031 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8161f52-83e9-4a81-9930-8f18d3b388c4 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:02.125128 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b8161f52-83e9-4a81-9930-8f18d3b388c4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6feee69ec37946468c1929dd560a3190', 'request_id': 'req-b8161f52-83e9-4a81-9930-8f18d3b388c4'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:02.134819 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b8161f52-83e9-4a81-9930-8f18d3b388c4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6feee69ec37946468c1929dd560a3190', 'request_id': 'req-b8161f52-83e9-4a81-9930-8f18d3b388c4'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:02.141589 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b8161f52-83e9-4a81-9930-8f18d3b388c4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6feee69ec37946468c1929dd560a3190', 'request_id': 'req-b8161f52-83e9-4a81-9930-8f18d3b388c4'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:02.154117 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b8161f52-83e9-4a81-9930-8f18d3b388c4 admin admin] The token cache is being invalidate because project 6feee69ec37946468c1929dd560a3190 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:06:02.154531 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b8161f52-83e9-4a81-9930-8f18d3b388c4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:02.155225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b8161f52-83e9-4a81-9930-8f18d3b388c4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:02.156781 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 989/1965] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:06:01 2022] DELETE /identity/v3/projects/6feee69ec37946468c1929dd560a3190 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:02.161939 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e29f08-cdd2-4b63-952e-e07690768b03 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:02.162217 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e29f08-cdd2-4b63-952e-e07690768b03 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:02.162485 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e29f08-cdd2-4b63-952e-e07690768b03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:02.185250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-40e29f08-cdd2-4b63-952e-e07690768b03 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:06:02.186502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-40e29f08-cdd2-4b63-952e-e07690768b03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:02.264928 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 977/1966] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:06:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:06:02.273511 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28798a70-031f-4770-a6b8-836aaabdd3ed None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:02.273793 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28798a70-031f-4770-a6b8-836aaabdd3ed None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:02.274062 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28798a70-031f-4770-a6b8-836aaabdd3ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:02.294058 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-28798a70-031f-4770-a6b8-836aaabdd3ed None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:06:02.295256 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-28798a70-031f-4770-a6b8-836aaabdd3ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:02.303355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 990/1967] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:06:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:06:02.308099 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e29f08-cdd2-4b63-952e-e07690768b03 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:02.318084 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e29f08-cdd2-4b63-952e-e07690768b03 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:02.323606 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec31615e-c1dc-49ee-a3e2-aeee3b2690b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:02.324371 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec31615e-c1dc-49ee-a3e2-aeee3b2690b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:02.324563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec31615e-c1dc-49ee-a3e2-aeee3b2690b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:02.324832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec31615e-c1dc-49ee-a3e2-aeee3b2690b4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:02.325486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec31615e-c1dc-49ee-a3e2-aeee3b2690b4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:06:02.325784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec31615e-c1dc-49ee-a3e2-aeee3b2690b4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:02.327382 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec31615e-c1dc-49ee-a3e2-aeee3b2690b4 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:02.339784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 978/1968] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:06:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 01:06:02.344382 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28798a70-031f-4770-a6b8-836aaabdd3ed None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:02.354012 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28798a70-031f-4770-a6b8-836aaabdd3ed None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:02.358791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309ad699-2623-4a32-abb5-25de3073b19e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:02.359385 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309ad699-2623-4a32-abb5-25de3073b19e admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:02.359655 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309ad699-2623-4a32-abb5-25de3073b19e admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:02.359956 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309ad699-2623-4a32-abb5-25de3073b19e admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:02.360590 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309ad699-2623-4a32-abb5-25de3073b19e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:02.362495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309ad699-2623-4a32-abb5-25de3073b19e admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:02.391925 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 991/1969] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:06:02 2022] POST /identity/v3/projects => generated 367 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:06:02.396585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec31615e-c1dc-49ee-a3e2-aeee3b2690b4 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:02.407038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec31615e-c1dc-49ee-a3e2-aeee3b2690b4 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:02.452414 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:02.466257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8d55bb-e588-4bdc-953b-1d180645ecef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:02.466863 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8d55bb-e588-4bdc-953b-1d180645ecef admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:02.467143 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8d55bb-e588-4bdc-953b-1d180645ecef admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:02.467407 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8d55bb-e588-4bdc-953b-1d180645ecef admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:02.468076 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8d55bb-e588-4bdc-953b-1d180645ecef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:02.469999 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8d55bb-e588-4bdc-953b-1d180645ecef admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:02.495599 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 979/1970] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:06:02 2022] POST /identity/v3/users => generated 284 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:06:02.500084 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309ad699-2623-4a32-abb5-25de3073b19e admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:02.513602 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309ad699-2623-4a32-abb5-25de3073b19e admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:02.519166 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314873c8-b0f3-441a-9c9e-9f77e6657886 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:02.519798 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314873c8-b0f3-441a-9c9e-9f77e6657886 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:02.520072 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314873c8-b0f3-441a-9c9e-9f77e6657886 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:02.520339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314873c8-b0f3-441a-9c9e-9f77e6657886 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:02.521025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314873c8-b0f3-441a-9c9e-9f77e6657886 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:02.522620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314873c8-b0f3-441a-9c9e-9f77e6657886 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:02.532538 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 992/1971] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:06:02 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:06:02.536998 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8d55bb-e588-4bdc-953b-1d180645ecef admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:02.547491 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8d55bb-e588-4bdc-953b-1d180645ecef admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:02.553008 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90902dd-d43d-4633-a37c-a4982a2fb5ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:02.553611 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90902dd-d43d-4633-a37c-a4982a2fb5ba admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:02.553909 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90902dd-d43d-4633-a37c-a4982a2fb5ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:02.554175 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90902dd-d43d-4633-a37c-a4982a2fb5ba admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:02.554788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90902dd-d43d-4633-a37c-a4982a2fb5ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:02.556413 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90902dd-d43d-4633-a37c-a4982a2fb5ba admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:02.567161 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 980/1972] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:06:02 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:06:02.571274 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314873c8-b0f3-441a-9c9e-9f77e6657886 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:02.581595 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314873c8-b0f3-441a-9c9e-9f77e6657886 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:02.587149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab9ad54-5434-40ce-b4d2-4c615c1bf483 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:02.587848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab9ad54-5434-40ce-b4d2-4c615c1bf483 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:02.588207 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab9ad54-5434-40ce-b4d2-4c615c1bf483 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:02.588832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab9ad54-5434-40ce-b4d2-4c615c1bf483 admin admin] PATH_INFO: `/v3/projects/f4fb1d16e027445f9542b13952fbba15/users/82ee31fe16464265a375e6cce615ce5b/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:02.606082 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab9ad54-5434-40ce-b4d2-4c615c1bf483 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f4fb1d16e027445f9542b13952fbba15, user_id=82ee31fe16464265a375e6cce615ce5b, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:02.608785 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab9ad54-5434-40ce-b4d2-4c615c1bf483 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:02.624266 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 993/1973] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:06:02 2022] PUT /identity/v3/projects/f4fb1d16e027445f9542b13952fbba15/users/82ee31fe16464265a375e6cce615ce5b/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:06:02.636151 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7cb2c2-89bf-43eb-988c-0169fb2b77da None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:02.636408 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7cb2c2-89bf-43eb-988c-0169fb2b77da None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:02.636770 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7cb2c2-89bf-43eb-988c-0169fb2b77da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:02.665512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-0e7cb2c2-89bf-43eb-988c-0169fb2b77da None None] MFA Rules not processed for user `82ee31fe16464265a375e6cce615ce5b`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:06:02.666682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-0e7cb2c2-89bf-43eb-988c-0169fb2b77da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:02.733884 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 981/1974] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:06:02 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:06:02.738458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab9ad54-5434-40ce-b4d2-4c615c1bf483 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:02.753058 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:02.822360 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab9ad54-5434-40ce-b4d2-4c615c1bf483 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:02.827776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a14c9a-7446-43c2-9b84-1c01ead600c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:02.828447 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a14c9a-7446-43c2-9b84-1c01ead600c9 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:02.828724 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a14c9a-7446-43c2-9b84-1c01ead600c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:02.828990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a14c9a-7446-43c2-9b84-1c01ead600c9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:02.829626 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a14c9a-7446-43c2-9b84-1c01ead600c9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:02.831666 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a14c9a-7446-43c2-9b84-1c01ead600c9 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:02.871520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 994/1975] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:06:02 2022] POST /identity/v3/projects => generated 367 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:06:02.872761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7cb2c2-89bf-43eb-988c-0169fb2b77da None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:02.884499 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7cb2c2-89bf-43eb-988c-0169fb2b77da None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:02.948356 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765b2a33-ec77-4ba2-89b8-d57834f77ac4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:02.948999 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765b2a33-ec77-4ba2-89b8-d57834f77ac4 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:02.949271 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765b2a33-ec77-4ba2-89b8-d57834f77ac4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:02.949533 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765b2a33-ec77-4ba2-89b8-d57834f77ac4 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:02.950168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-765b2a33-ec77-4ba2-89b8-d57834f77ac4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:02.952177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-765b2a33-ec77-4ba2-89b8-d57834f77ac4 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:02.976289 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 982/1976] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:06:02 2022] POST /identity/v3/users => generated 290 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:06:02.981890 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a14c9a-7446-43c2-9b84-1c01ead600c9 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:02.992714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a14c9a-7446-43c2-9b84-1c01ead600c9 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:02.998032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49650b9-ccbd-404a-8030-e62b56df9a27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:02.998739 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49650b9-ccbd-404a-8030-e62b56df9a27 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:02.999055 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49650b9-ccbd-404a-8030-e62b56df9a27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:02.999388 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49650b9-ccbd-404a-8030-e62b56df9a27 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:03.000083 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49650b9-ccbd-404a-8030-e62b56df9a27 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:03.001676 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49650b9-ccbd-404a-8030-e62b56df9a27 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:03.013006 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 995/1977] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:06:02 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:06:03.017473 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765b2a33-ec77-4ba2-89b8-d57834f77ac4 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:03.028444 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765b2a33-ec77-4ba2-89b8-d57834f77ac4 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:03.034272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb45b2c3-467c-4316-bd8f-52e3ab0af07e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:03.034923 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb45b2c3-467c-4316-bd8f-52e3ab0af07e admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:03.035259 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb45b2c3-467c-4316-bd8f-52e3ab0af07e admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:03.035528 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb45b2c3-467c-4316-bd8f-52e3ab0af07e admin admin] PATH_INFO: `/v3/projects/bd52789b479c40d684dcfb6191ac9dc7/users/0a2541867e5043758fa2f7e30db24a85/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:03.051857 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb45b2c3-467c-4316-bd8f-52e3ab0af07e admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd52789b479c40d684dcfb6191ac9dc7, user_id=0a2541867e5043758fa2f7e30db24a85, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:03.054865 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb45b2c3-467c-4316-bd8f-52e3ab0af07e admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:03.070745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 983/1978] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:06:03 2022] PUT /identity/v3/projects/bd52789b479c40d684dcfb6191ac9dc7/users/0a2541867e5043758fa2f7e30db24a85/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:03.076881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49650b9-ccbd-404a-8030-e62b56df9a27 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:03.088409 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49650b9-ccbd-404a-8030-e62b56df9a27 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:03.149678 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e34687-0ddf-4a00-84a9-eb224dfa3e58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:03.150286 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e34687-0ddf-4a00-84a9-eb224dfa3e58 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:03.151146 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e34687-0ddf-4a00-84a9-eb224dfa3e58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:03.151146 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e34687-0ddf-4a00-84a9-eb224dfa3e58 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:03.151459 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e34687-0ddf-4a00-84a9-eb224dfa3e58 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:03.153117 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e34687-0ddf-4a00-84a9-eb224dfa3e58 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:03.163200 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 996/1979] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:06:03 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:06:03.167643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb45b2c3-467c-4316-bd8f-52e3ab0af07e admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:03.178051 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb45b2c3-467c-4316-bd8f-52e3ab0af07e admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:03.183550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a44e145-cf19-47e8-a6a4-31d11009a45a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:03.184472 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a44e145-cf19-47e8-a6a4-31d11009a45a admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:03.184764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a44e145-cf19-47e8-a6a4-31d11009a45a admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:03.185287 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a44e145-cf19-47e8-a6a4-31d11009a45a admin admin] PATH_INFO: `/v3/projects/bd52789b479c40d684dcfb6191ac9dc7/users/0a2541867e5043758fa2f7e30db24a85/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:03.187690 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a44e145-cf19-47e8-a6a4-31d11009a45a admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd52789b479c40d684dcfb6191ac9dc7, user_id=0a2541867e5043758fa2f7e30db24a85, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:03.190889 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a44e145-cf19-47e8-a6a4-31d11009a45a admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:03.202370 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 984/1980] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:06:03 2022] PUT /identity/v3/projects/bd52789b479c40d684dcfb6191ac9dc7/users/0a2541867e5043758fa2f7e30db24a85/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:03.213135 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414cfcef-6444-45d5-9977-c0195d991f9f None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:03.213413 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414cfcef-6444-45d5-9977-c0195d991f9f None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:03.213990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414cfcef-6444-45d5-9977-c0195d991f9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:03.245146 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-414cfcef-6444-45d5-9977-c0195d991f9f None None] MFA Rules not processed for user `0a2541867e5043758fa2f7e30db24a85`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:06:03.246385 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-414cfcef-6444-45d5-9977-c0195d991f9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:03.341911 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 997/1981] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:06:03 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:06:03.353937 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a44e145-cf19-47e8-a6a4-31d11009a45a admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:03.355748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-8a44e145-cf19-47e8-a6a4-31d11009a45a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:03.372228 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:03.425191 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a44e145-cf19-47e8-a6a4-31d11009a45a admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:03.429051 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2ab2b3-440b-4ceb-b1de-07e189fe90dd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:03.429665 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2ab2b3-440b-4ceb-b1de-07e189fe90dd service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:03.429934 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2ab2b3-440b-4ceb-b1de-07e189fe90dd service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:03.430194 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2ab2b3-440b-4ceb-b1de-07e189fe90dd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:03.439896 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2ab2b3-440b-4ceb-b1de-07e189fe90dd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:03.441246 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2ab2b3-440b-4ceb-b1de-07e189fe90dd service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:03.451302 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 985/1982] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:06:03 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:06:03.501256 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414cfcef-6444-45d5-9977-c0195d991f9f None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:03.502643 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-414cfcef-6444-45d5-9977-c0195d991f9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:03.514457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414cfcef-6444-45d5-9977-c0195d991f9f None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:03.518044 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35681d70-49c1-4798-8457-f5e7924800bb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:03.518647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35681d70-49c1-4798-8457-f5e7924800bb service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:03.518919 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35681d70-49c1-4798-8457-f5e7924800bb service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:03.519181 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35681d70-49c1-4798-8457-f5e7924800bb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:03.523367 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:03.576305 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35681d70-49c1-4798-8457-f5e7924800bb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:03.577730 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35681d70-49c1-4798-8457-f5e7924800bb service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:03.586997 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 998/1983] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:06:03 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:06:06.978523 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2ab2b3-440b-4ceb-b1de-07e189fe90dd service cinder] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:07.040717 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2ab2b3-440b-4ceb-b1de-07e189fe90dd service cinder] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:07.045908 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8bd418b-c779-477f-99a6-ddc9cc198c73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:07.046543 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bd418b-c779-477f-99a6-ddc9cc198c73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:07.046811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bd418b-c779-477f-99a6-ddc9cc198c73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:07.047074 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bd418b-c779-477f-99a6-ddc9cc198c73 admin admin] PATH_INFO: `/v3/users/82ee31fe16464265a375e6cce615ce5b` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:07.048369 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8bd418b-c779-477f-99a6-ddc9cc198c73 admin admin] RBAC: Authorizing `identity:delete_user(user_id=82ee31fe16464265a375e6cce615ce5b)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:07.050353 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8bd418b-c779-477f-99a6-ddc9cc198c73 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:07.114889 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a8bd418b-c779-477f-99a6-ddc9cc198c73 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '82ee31fe16464265a375e6cce615ce5b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:07.126260 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a8bd418b-c779-477f-99a6-ddc9cc198c73 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '82ee31fe16464265a375e6cce615ce5b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:07.127218 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a8bd418b-c779-477f-99a6-ddc9cc198c73 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '82ee31fe16464265a375e6cce615ce5b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:07.138442 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a8bd418b-c779-477f-99a6-ddc9cc198c73 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '82ee31fe16464265a375e6cce615ce5b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:07.154578 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-a8bd418b-c779-477f-99a6-ddc9cc198c73 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '82ee31fe16464265a375e6cce615ce5b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:07.160712 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 986/1984] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:06:06 2022] DELETE /identity/v3/users/82ee31fe16464265a375e6cce615ce5b => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:07.165555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62cd8ad-d85c-43d7-998d-14cbb97b88a3 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:07.166016 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62cd8ad-d85c-43d7-998d-14cbb97b88a3 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:07.166115 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62cd8ad-d85c-43d7-998d-14cbb97b88a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:07.186009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-e62cd8ad-d85c-43d7-998d-14cbb97b88a3 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:06:07.187346 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e62cd8ad-d85c-43d7-998d-14cbb97b88a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:07.249321 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 999/1985] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:06:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:06:07.259647 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8bd418b-c779-477f-99a6-ddc9cc198c73 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:07.261269 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a8bd418b-c779-477f-99a6-ddc9cc198c73 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:07.295848 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:07.321528 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8bd418b-c779-477f-99a6-ddc9cc198c73 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:07.325536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2def16-ea79-4bd0-83c3-3f36ebabb1da service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:07.326143 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2def16-ea79-4bd0-83c3-3f36ebabb1da service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:07.326391 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2def16-ea79-4bd0-83c3-3f36ebabb1da service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:07.326685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2def16-ea79-4bd0-83c3-3f36ebabb1da service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:07.336581 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2def16-ea79-4bd0-83c3-3f36ebabb1da service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:07.338042 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2def16-ea79-4bd0-83c3-3f36ebabb1da service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:07.348820 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 987/1986] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:06:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:06:07.715464 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62cd8ad-d85c-43d7-998d-14cbb97b88a3 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:07.717051 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e62cd8ad-d85c-43d7-998d-14cbb97b88a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:07.729514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62cd8ad-d85c-43d7-998d-14cbb97b88a3 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:07.734150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defffd72-716e-413d-a5f1-3c9cd9eb31a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:07.734806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defffd72-716e-413d-a5f1-3c9cd9eb31a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:07.735081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defffd72-716e-413d-a5f1-3c9cd9eb31a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:07.735339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defffd72-716e-413d-a5f1-3c9cd9eb31a5 admin admin] PATH_INFO: `/v3/projects/f4fb1d16e027445f9542b13952fbba15` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:07.736494 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defffd72-716e-413d-a5f1-3c9cd9eb31a5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f4fb1d16e027445f9542b13952fbba15)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:07.738379 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defffd72-716e-413d-a5f1-3c9cd9eb31a5 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:07.788530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-defffd72-716e-413d-a5f1-3c9cd9eb31a5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f4fb1d16e027445f9542b13952fbba15', 'request_id': 'req-defffd72-716e-413d-a5f1-3c9cd9eb31a5'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:07.797561 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-defffd72-716e-413d-a5f1-3c9cd9eb31a5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f4fb1d16e027445f9542b13952fbba15', 'request_id': 'req-defffd72-716e-413d-a5f1-3c9cd9eb31a5'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:07.803538 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-defffd72-716e-413d-a5f1-3c9cd9eb31a5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f4fb1d16e027445f9542b13952fbba15', 'request_id': 'req-defffd72-716e-413d-a5f1-3c9cd9eb31a5'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:07.815836 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-defffd72-716e-413d-a5f1-3c9cd9eb31a5 admin admin] The token cache is being invalidate because project f4fb1d16e027445f9542b13952fbba15 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:06:07.816120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-defffd72-716e-413d-a5f1-3c9cd9eb31a5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:07.816810 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-defffd72-716e-413d-a5f1-3c9cd9eb31a5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:07.818423 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1000/1987] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:06:07 2022] DELETE /identity/v3/projects/f4fb1d16e027445f9542b13952fbba15 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:07.822516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2def16-ea79-4bd0-83c3-3f36ebabb1da service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:07.832417 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2def16-ea79-4bd0-83c3-3f36ebabb1da service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:07.892445 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cdfd1d-359a-4d4d-9ac3-70e7bba84253 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:07.893072 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cdfd1d-359a-4d4d-9ac3-70e7bba84253 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:07.893340 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cdfd1d-359a-4d4d-9ac3-70e7bba84253 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:07.893600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cdfd1d-359a-4d4d-9ac3-70e7bba84253 admin admin] PATH_INFO: `/v3/users/0a2541867e5043758fa2f7e30db24a85` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:07.894782 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82cdfd1d-359a-4d4d-9ac3-70e7bba84253 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0a2541867e5043758fa2f7e30db24a85)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:07.896908 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82cdfd1d-359a-4d4d-9ac3-70e7bba84253 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:07.948189 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-82cdfd1d-359a-4d4d-9ac3-70e7bba84253 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0a2541867e5043758fa2f7e30db24a85'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:07.958807 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-82cdfd1d-359a-4d4d-9ac3-70e7bba84253 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0a2541867e5043758fa2f7e30db24a85'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:07.959422 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-82cdfd1d-359a-4d4d-9ac3-70e7bba84253 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0a2541867e5043758fa2f7e30db24a85'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:07.967210 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-82cdfd1d-359a-4d4d-9ac3-70e7bba84253 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0a2541867e5043758fa2f7e30db24a85'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:07.980533 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-82cdfd1d-359a-4d4d-9ac3-70e7bba84253 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0a2541867e5043758fa2f7e30db24a85'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:07.990885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 988/1988] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:06:07 2022] DELETE /identity/v3/users/0a2541867e5043758fa2f7e30db24a85 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:08.461081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defffd72-716e-413d-a5f1-3c9cd9eb31a5 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:08.462574 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-defffd72-716e-413d-a5f1-3c9cd9eb31a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:08.476513 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defffd72-716e-413d-a5f1-3c9cd9eb31a5 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:08.514309 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:08.547708 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101b8160-649f-4f77-87db-b60506a0d979 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:08.548384 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101b8160-649f-4f77-87db-b60506a0d979 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:08.548657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101b8160-649f-4f77-87db-b60506a0d979 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:08.548921 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101b8160-649f-4f77-87db-b60506a0d979 admin admin] PATH_INFO: `/v3/projects/bd52789b479c40d684dcfb6191ac9dc7` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:08.550023 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101b8160-649f-4f77-87db-b60506a0d979 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd52789b479c40d684dcfb6191ac9dc7)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:08.552062 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101b8160-649f-4f77-87db-b60506a0d979 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:08.601275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-101b8160-649f-4f77-87db-b60506a0d979 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd52789b479c40d684dcfb6191ac9dc7', 'request_id': 'req-101b8160-649f-4f77-87db-b60506a0d979'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:08.611208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-101b8160-649f-4f77-87db-b60506a0d979 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd52789b479c40d684dcfb6191ac9dc7', 'request_id': 'req-101b8160-649f-4f77-87db-b60506a0d979'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:08.617131 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-101b8160-649f-4f77-87db-b60506a0d979 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd52789b479c40d684dcfb6191ac9dc7', 'request_id': 'req-101b8160-649f-4f77-87db-b60506a0d979'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:08.628833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-101b8160-649f-4f77-87db-b60506a0d979 admin admin] The token cache is being invalidate because project bd52789b479c40d684dcfb6191ac9dc7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:06:08.629121 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-101b8160-649f-4f77-87db-b60506a0d979 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:08.629756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-101b8160-649f-4f77-87db-b60506a0d979 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:08.631382 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1001/1989] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:06:08 2022] DELETE /identity/v3/projects/bd52789b479c40d684dcfb6191ac9dc7 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:08.636435 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e80476-9a66-4af3-b92c-8806e1bd9354 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:08.636740 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e80476-9a66-4af3-b92c-8806e1bd9354 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:08.637007 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e80476-9a66-4af3-b92c-8806e1bd9354 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:08.657426 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-75e80476-9a66-4af3-b92c-8806e1bd9354 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:06:08.658658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-75e80476-9a66-4af3-b92c-8806e1bd9354 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:08.709854 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:08.727140 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 989/1990] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:06:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:06:08.736246 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6f3af0-0255-424c-8e2c-d863033eb49c None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:08.736529 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6f3af0-0255-424c-8e2c-d863033eb49c None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:08.737349 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6f3af0-0255-424c-8e2c-d863033eb49c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:08.757870 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-2c6f3af0-0255-424c-8e2c-d863033eb49c None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:06:08.759083 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-2c6f3af0-0255-424c-8e2c-d863033eb49c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:08.767498 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1002/1991] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:06:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:06:08.771808 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e80476-9a66-4af3-b92c-8806e1bd9354 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:08.781703 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e80476-9a66-4af3-b92c-8806e1bd9354 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:08.787173 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43719532-b8a8-4f2c-bf42-aa87dfe6b7ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:08.787817 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43719532-b8a8-4f2c-bf42-aa87dfe6b7ac admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:08.788097 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43719532-b8a8-4f2c-bf42-aa87dfe6b7ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:08.788409 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43719532-b8a8-4f2c-bf42-aa87dfe6b7ac admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:08.789057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43719532-b8a8-4f2c-bf42-aa87dfe6b7ac admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:06:08.789526 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43719532-b8a8-4f2c-bf42-aa87dfe6b7ac admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:08.791096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43719532-b8a8-4f2c-bf42-aa87dfe6b7ac admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:08.803950 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 990/1992] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:06:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 01:06:08.808429 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6f3af0-0255-424c-8e2c-d863033eb49c None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:08.818034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6f3af0-0255-424c-8e2c-d863033eb49c None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:08.823303 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507c25da-a2d2-4247-b785-ffb5e1e5a04c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:08.824136 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507c25da-a2d2-4247-b785-ffb5e1e5a04c admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:08.824601 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507c25da-a2d2-4247-b785-ffb5e1e5a04c admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:08.824873 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507c25da-a2d2-4247-b785-ffb5e1e5a04c admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:08.825539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507c25da-a2d2-4247-b785-ffb5e1e5a04c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:08.827418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507c25da-a2d2-4247-b785-ffb5e1e5a04c admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:08.858824 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1003/1993] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:06:08 2022] POST /identity/v3/projects => generated 399 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:06:08.863168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43719532-b8a8-4f2c-bf42-aa87dfe6b7ac admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:08.873268 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43719532-b8a8-4f2c-bf42-aa87dfe6b7ac admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:08.928757 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc06cd6-7289-4955-8480-2aabe117fe13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:08.929384 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc06cd6-7289-4955-8480-2aabe117fe13 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:08.929656 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc06cd6-7289-4955-8480-2aabe117fe13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:08.929923 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc06cd6-7289-4955-8480-2aabe117fe13 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:08.930577 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc06cd6-7289-4955-8480-2aabe117fe13 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:08.932560 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc06cd6-7289-4955-8480-2aabe117fe13 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:08.955964 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 991/1994] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:06:08 2022] POST /identity/v3/users => generated 300 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:06:08.960736 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507c25da-a2d2-4247-b785-ffb5e1e5a04c admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:08.971696 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507c25da-a2d2-4247-b785-ffb5e1e5a04c admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:08.977056 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc0724f-a942-4ced-adb8-b6b4dd6778ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:08.978020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc0724f-a942-4ced-adb8-b6b4dd6778ca admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:08.978236 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc0724f-a942-4ced-adb8-b6b4dd6778ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:08.978535 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc0724f-a942-4ced-adb8-b6b4dd6778ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:08.979209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc0724f-a942-4ced-adb8-b6b4dd6778ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:08.980821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc0724f-a942-4ced-adb8-b6b4dd6778ca admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:08.991893 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1004/1995] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:06:08 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:06:08.996151 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc06cd6-7289-4955-8480-2aabe117fe13 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:09.006973 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc06cd6-7289-4955-8480-2aabe117fe13 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:09.012613 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4f3905-2ba9-4714-a81a-5a44d54e91eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:09.013238 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4f3905-2ba9-4714-a81a-5a44d54e91eb admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:09.013524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4f3905-2ba9-4714-a81a-5a44d54e91eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:09.013782 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4f3905-2ba9-4714-a81a-5a44d54e91eb admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:09.014420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e4f3905-2ba9-4714-a81a-5a44d54e91eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:09.016051 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e4f3905-2ba9-4714-a81a-5a44d54e91eb admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:09.027251 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 992/1996] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:06:08 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:06:09.031596 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc0724f-a942-4ced-adb8-b6b4dd6778ca admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:09.042452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc0724f-a942-4ced-adb8-b6b4dd6778ca admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:09.047889 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac8d6c9-b5af-455a-b199-9cb5d39c1877 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:09.048513 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac8d6c9-b5af-455a-b199-9cb5d39c1877 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:09.048786 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac8d6c9-b5af-455a-b199-9cb5d39c1877 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:09.049054 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac8d6c9-b5af-455a-b199-9cb5d39c1877 admin admin] PATH_INFO: `/v3/projects/7c4e4351375047cbab715382d6dee35c/users/028542cf023644508a1a0c5e62e7203f/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:09.063378 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac8d6c9-b5af-455a-b199-9cb5d39c1877 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7c4e4351375047cbab715382d6dee35c, user_id=028542cf023644508a1a0c5e62e7203f, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:09.066039 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac8d6c9-b5af-455a-b199-9cb5d39c1877 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:09.081087 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1005/1997] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:06:09 2022] PUT /identity/v3/projects/7c4e4351375047cbab715382d6dee35c/users/028542cf023644508a1a0c5e62e7203f/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:09.092305 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f1868b-419b-41c4-b282-21be7252c69d None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:09.092587 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f1868b-419b-41c4-b282-21be7252c69d None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:09.092856 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f1868b-419b-41c4-b282-21be7252c69d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:09.121722 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-43f1868b-419b-41c4-b282-21be7252c69d None None] MFA Rules not processed for user `028542cf023644508a1a0c5e62e7203f`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:06:09.122980 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-43f1868b-419b-41c4-b282-21be7252c69d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:09.190993 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 993/1998] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:06:09 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:06:09.195992 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac8d6c9-b5af-455a-b199-9cb5d39c1877 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:09.256434 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:09.261430 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac8d6c9-b5af-455a-b199-9cb5d39c1877 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:09.267297 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5118b3a8-2d82-421f-9856-15ead62e9fe4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:09.268104 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5118b3a8-2d82-421f-9856-15ead62e9fe4 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:09.268416 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5118b3a8-2d82-421f-9856-15ead62e9fe4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:09.268834 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5118b3a8-2d82-421f-9856-15ead62e9fe4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:09.269676 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5118b3a8-2d82-421f-9856-15ead62e9fe4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:09.272260 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5118b3a8-2d82-421f-9856-15ead62e9fe4 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:09.301775 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1006/1999] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:06:09 2022] POST /identity/v3/projects => generated 399 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:06:09.306164 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f1868b-419b-41c4-b282-21be7252c69d None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:09.317544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f1868b-419b-41c4-b282-21be7252c69d None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:09.370429 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:09.384124 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e693217a-0377-4c58-bc8b-d3ff666bdd29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:09.384773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e693217a-0377-4c58-bc8b-d3ff666bdd29 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:09.385043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e693217a-0377-4c58-bc8b-d3ff666bdd29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:09.385286 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e693217a-0377-4c58-bc8b-d3ff666bdd29 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:09.385919 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e693217a-0377-4c58-bc8b-d3ff666bdd29 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:09.387861 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e693217a-0377-4c58-bc8b-d3ff666bdd29 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:09.410326 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 994/2000] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:06:09 2022] POST /identity/v3/users => generated 306 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:06:09.414619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5118b3a8-2d82-421f-9856-15ead62e9fe4 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:09.424817 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5118b3a8-2d82-421f-9856-15ead62e9fe4 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:09.429625 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafe28bf-ad23-4b3a-b857-18c2a7103b6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:09.430226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafe28bf-ad23-4b3a-b857-18c2a7103b6f admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:09.430530 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafe28bf-ad23-4b3a-b857-18c2a7103b6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:09.430799 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafe28bf-ad23-4b3a-b857-18c2a7103b6f admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:09.431433 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bafe28bf-ad23-4b3a-b857-18c2a7103b6f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:09.433094 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bafe28bf-ad23-4b3a-b857-18c2a7103b6f admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:09.444197 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1007/2001] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:06:09 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:06:09.448348 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e693217a-0377-4c58-bc8b-d3ff666bdd29 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:09.460745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e693217a-0377-4c58-bc8b-d3ff666bdd29 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:09.465941 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302d324d-9dde-4674-af79-a000287887f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:09.466620 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302d324d-9dde-4674-af79-a000287887f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:09.466900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302d324d-9dde-4674-af79-a000287887f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:09.467171 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302d324d-9dde-4674-af79-a000287887f3 admin admin] PATH_INFO: `/v3/projects/c6f1dec453734393a99fd075094ebde7/users/6d032352f0f14f84ba148218aeef92bc/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:09.482655 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-302d324d-9dde-4674-af79-a000287887f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6f1dec453734393a99fd075094ebde7, user_id=6d032352f0f14f84ba148218aeef92bc, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:09.485254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-302d324d-9dde-4674-af79-a000287887f3 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:09.500823 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 995/2002] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:06:09 2022] PUT /identity/v3/projects/c6f1dec453734393a99fd075094ebde7/users/6d032352f0f14f84ba148218aeef92bc/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:09.505141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafe28bf-ad23-4b3a-b857-18c2a7103b6f admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:09.516472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafe28bf-ad23-4b3a-b857-18c2a7103b6f admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:09.575192 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87932942-ec2e-451b-a71c-b1a006f23ded admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:09.575834 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87932942-ec2e-451b-a71c-b1a006f23ded admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:09.576236 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87932942-ec2e-451b-a71c-b1a006f23ded admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:09.576507 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87932942-ec2e-451b-a71c-b1a006f23ded admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:09.577139 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87932942-ec2e-451b-a71c-b1a006f23ded admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:09.578722 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87932942-ec2e-451b-a71c-b1a006f23ded admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:09.589187 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1008/2003] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:06:09 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:06:09.593990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302d324d-9dde-4674-af79-a000287887f3 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:09.605609 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302d324d-9dde-4674-af79-a000287887f3 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:09.611032 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec4bc35-4c48-4b9b-a9ca-7a86653449c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:09.611685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec4bc35-4c48-4b9b-a9ca-7a86653449c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:09.612075 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec4bc35-4c48-4b9b-a9ca-7a86653449c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:09.612341 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec4bc35-4c48-4b9b-a9ca-7a86653449c6 admin admin] PATH_INFO: `/v3/projects/c6f1dec453734393a99fd075094ebde7/users/6d032352f0f14f84ba148218aeef92bc/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:09.614383 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec4bc35-4c48-4b9b-a9ca-7a86653449c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6f1dec453734393a99fd075094ebde7, user_id=6d032352f0f14f84ba148218aeef92bc, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:09.617357 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec4bc35-4c48-4b9b-a9ca-7a86653449c6 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:09.628584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 996/2004] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:06:09 2022] PUT /identity/v3/projects/c6f1dec453734393a99fd075094ebde7/users/6d032352f0f14f84ba148218aeef92bc/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:09.639669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1c900c-2195-4113-93e4-71602ec01e8f None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:09.639986 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1c900c-2195-4113-93e4-71602ec01e8f None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:09.640258 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1c900c-2195-4113-93e4-71602ec01e8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:09.669164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-4b1c900c-2195-4113-93e4-71602ec01e8f None None] MFA Rules not processed for user `6d032352f0f14f84ba148218aeef92bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:06:09.670498 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-4b1c900c-2195-4113-93e4-71602ec01e8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:09.746361 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1009/2005] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:06:09 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:06:09.758994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec4bc35-4c48-4b9b-a9ca-7a86653449c6 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:09.761095 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-3ec4bc35-4c48-4b9b-a9ca-7a86653449c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:09.825303 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec4bc35-4c48-4b9b-a9ca-7a86653449c6 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:09.829109 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2e576e-798c-475d-9aad-edad8da94f3c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:09.829717 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2e576e-798c-475d-9aad-edad8da94f3c service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:09.829988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2e576e-798c-475d-9aad-edad8da94f3c service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:09.830256 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2e576e-798c-475d-9aad-edad8da94f3c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:09.839889 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2e576e-798c-475d-9aad-edad8da94f3c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:09.841314 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2e576e-798c-475d-9aad-edad8da94f3c service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:09.850709 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 997/2006] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:06:09 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:06:09.941935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1c900c-2195-4113-93e4-71602ec01e8f None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:09.952300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1c900c-2195-4113-93e4-71602ec01e8f None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:09.955870 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d1066e-d6d4-46a8-b2e6-6569b0d04707 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:09.956484 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d1066e-d6d4-46a8-b2e6-6569b0d04707 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:09.956786 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d1066e-d6d4-46a8-b2e6-6569b0d04707 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:09.957240 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d1066e-d6d4-46a8-b2e6-6569b0d04707 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:10.008528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:10.020627 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d1066e-d6d4-46a8-b2e6-6569b0d04707 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:10.022592 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d1066e-d6d4-46a8-b2e6-6569b0d04707 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:10.036418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1010/2007] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:06:09 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:06:12.762968 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2e576e-798c-475d-9aad-edad8da94f3c service cinder] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:12.764631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-8c2e576e-798c-475d-9aad-edad8da94f3c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:12.852501 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2e576e-798c-475d-9aad-edad8da94f3c service cinder] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:12.858435 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256f826e-1fc4-4628-b545-d168cb6c0b3c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:12.859216 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256f826e-1fc4-4628-b545-d168cb6c0b3c service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:12.859451 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256f826e-1fc4-4628-b545-d168cb6c0b3c service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:12.859744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256f826e-1fc4-4628-b545-d168cb6c0b3c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:12.861771 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-256f826e-1fc4-4628-b545-d168cb6c0b3c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:12.873077 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256f826e-1fc4-4628-b545-d168cb6c0b3c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:12.874895 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256f826e-1fc4-4628-b545-d168cb6c0b3c service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:12.883320 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 998/2008] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:06:12 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:06:14.208701 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d1066e-d6d4-46a8-b2e6-6569b0d04707 service cinder] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:14.273310 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d1066e-d6d4-46a8-b2e6-6569b0d04707 service cinder] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:14.278246 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e51365-0458-46b3-a571-e2ce5fdc3e3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:14.278909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e51365-0458-46b3-a571-e2ce5fdc3e3b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:14.279186 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e51365-0458-46b3-a571-e2ce5fdc3e3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:14.279453 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e51365-0458-46b3-a571-e2ce5fdc3e3b admin admin] PATH_INFO: `/v3/users/028542cf023644508a1a0c5e62e7203f` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:14.280718 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e51365-0458-46b3-a571-e2ce5fdc3e3b admin admin] RBAC: Authorizing `identity:delete_user(user_id=028542cf023644508a1a0c5e62e7203f)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:14.282654 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e51365-0458-46b3-a571-e2ce5fdc3e3b admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:14.332625 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-85e51365-0458-46b3-a571-e2ce5fdc3e3b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '028542cf023644508a1a0c5e62e7203f'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:14.343426 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-85e51365-0458-46b3-a571-e2ce5fdc3e3b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '028542cf023644508a1a0c5e62e7203f'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:14.344631 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-85e51365-0458-46b3-a571-e2ce5fdc3e3b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '028542cf023644508a1a0c5e62e7203f'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:14.357241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-85e51365-0458-46b3-a571-e2ce5fdc3e3b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '028542cf023644508a1a0c5e62e7203f'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:14.364090 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-85e51365-0458-46b3-a571-e2ce5fdc3e3b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '028542cf023644508a1a0c5e62e7203f'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:14.370490 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1011/2009] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:06:14 2022] DELETE /identity/v3/users/028542cf023644508a1a0c5e62e7203f => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:14.375051 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df7c23d-a67b-4209-8f4d-4398acd20405 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:14.375511 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df7c23d-a67b-4209-8f4d-4398acd20405 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:14.375633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df7c23d-a67b-4209-8f4d-4398acd20405 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:14.395449 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-3df7c23d-a67b-4209-8f4d-4398acd20405 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:06:14.396913 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-3df7c23d-a67b-4209-8f4d-4398acd20405 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:14.438731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:14.455947 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 999/2010] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:06:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:06:14.467667 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e51365-0458-46b3-a571-e2ce5fdc3e3b admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:14.469276 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-85e51365-0458-46b3-a571-e2ce5fdc3e3b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:14.528944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e51365-0458-46b3-a571-e2ce5fdc3e3b admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:14.533252 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67de92d9-5b20-4df7-ac98-0d825bd50e44 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:14.533853 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67de92d9-5b20-4df7-ac98-0d825bd50e44 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:14.534124 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67de92d9-5b20-4df7-ac98-0d825bd50e44 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:14.534415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67de92d9-5b20-4df7-ac98-0d825bd50e44 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:14.544330 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67de92d9-5b20-4df7-ac98-0d825bd50e44 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:14.545773 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67de92d9-5b20-4df7-ac98-0d825bd50e44 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:14.556285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1012/2011] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:06:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 01:06:14.883695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df7c23d-a67b-4209-8f4d-4398acd20405 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:14.885270 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-3df7c23d-a67b-4209-8f4d-4398acd20405 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:14.899496 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df7c23d-a67b-4209-8f4d-4398acd20405 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:14.904550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715c1cd7-868e-4bf2-b821-d52c9a1c12dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:14.905251 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715c1cd7-868e-4bf2-b821-d52c9a1c12dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:14.905456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715c1cd7-868e-4bf2-b821-d52c9a1c12dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:14.905717 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715c1cd7-868e-4bf2-b821-d52c9a1c12dd admin admin] PATH_INFO: `/v3/projects/7c4e4351375047cbab715382d6dee35c` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:14.906835 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715c1cd7-868e-4bf2-b821-d52c9a1c12dd admin admin] RBAC: Authorizing `identity:delete_project(project_id=7c4e4351375047cbab715382d6dee35c)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:14.908798 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715c1cd7-868e-4bf2-b821-d52c9a1c12dd admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:14.957213 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-715c1cd7-868e-4bf2-b821-d52c9a1c12dd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7c4e4351375047cbab715382d6dee35c', 'request_id': 'req-715c1cd7-868e-4bf2-b821-d52c9a1c12dd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:14.962682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-715c1cd7-868e-4bf2-b821-d52c9a1c12dd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7c4e4351375047cbab715382d6dee35c', 'request_id': 'req-715c1cd7-868e-4bf2-b821-d52c9a1c12dd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:14.971919 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-715c1cd7-868e-4bf2-b821-d52c9a1c12dd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7c4e4351375047cbab715382d6dee35c', 'request_id': 'req-715c1cd7-868e-4bf2-b821-d52c9a1c12dd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:14.983273 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-715c1cd7-868e-4bf2-b821-d52c9a1c12dd admin admin] The token cache is being invalidate because project 7c4e4351375047cbab715382d6dee35c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:06:14.983553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-715c1cd7-868e-4bf2-b821-d52c9a1c12dd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:14.984217 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-715c1cd7-868e-4bf2-b821-d52c9a1c12dd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:14.985723 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1000/2012] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:06:14 2022] DELETE /identity/v3/projects/7c4e4351375047cbab715382d6dee35c => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:14.989827 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67de92d9-5b20-4df7-ac98-0d825bd50e44 service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:14.999527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67de92d9-5b20-4df7-ac98-0d825bd50e44 service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:15.007493 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:15.058745 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c2fe28-9d38-4d08-bcba-48fc4e438b77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:15.059376 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c2fe28-9d38-4d08-bcba-48fc4e438b77 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:15.059648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c2fe28-9d38-4d08-bcba-48fc4e438b77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:15.059971 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c2fe28-9d38-4d08-bcba-48fc4e438b77 admin admin] PATH_INFO: `/v3/users/6d032352f0f14f84ba148218aeef92bc` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:15.061093 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c2fe28-9d38-4d08-bcba-48fc4e438b77 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6d032352f0f14f84ba148218aeef92bc)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:15.062995 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c2fe28-9d38-4d08-bcba-48fc4e438b77 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:15.112440 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-14c2fe28-9d38-4d08-bcba-48fc4e438b77 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6d032352f0f14f84ba148218aeef92bc'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:15.121966 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-14c2fe28-9d38-4d08-bcba-48fc4e438b77 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6d032352f0f14f84ba148218aeef92bc'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:15.122585 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-14c2fe28-9d38-4d08-bcba-48fc4e438b77 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6d032352f0f14f84ba148218aeef92bc'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:15.134234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-14c2fe28-9d38-4d08-bcba-48fc4e438b77 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6d032352f0f14f84ba148218aeef92bc'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:15.140937 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-14c2fe28-9d38-4d08-bcba-48fc4e438b77 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6d032352f0f14f84ba148218aeef92bc'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:15.146732 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1013/2013] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:06:14 2022] DELETE /identity/v3/users/6d032352f0f14f84ba148218aeef92bc => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:15.445213 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715c1cd7-868e-4bf2-b821-d52c9a1c12dd admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:15.446803 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-715c1cd7-868e-4bf2-b821-d52c9a1c12dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:15.459954 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715c1cd7-868e-4bf2-b821-d52c9a1c12dd admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:15.520700 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1382f2f-2a66-4b60-8a9b-2f986dcd49cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:15.521374 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1382f2f-2a66-4b60-8a9b-2f986dcd49cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:15.521650 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1382f2f-2a66-4b60-8a9b-2f986dcd49cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:15.521913 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1382f2f-2a66-4b60-8a9b-2f986dcd49cd admin admin] PATH_INFO: `/v3/projects/c6f1dec453734393a99fd075094ebde7` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:15.523129 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1382f2f-2a66-4b60-8a9b-2f986dcd49cd admin admin] RBAC: Authorizing `identity:delete_project(project_id=c6f1dec453734393a99fd075094ebde7)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:15.525175 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1382f2f-2a66-4b60-8a9b-2f986dcd49cd admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:15.572283 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-c1382f2f-2a66-4b60-8a9b-2f986dcd49cd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c6f1dec453734393a99fd075094ebde7', 'request_id': 'req-c1382f2f-2a66-4b60-8a9b-2f986dcd49cd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:15.578210 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-c1382f2f-2a66-4b60-8a9b-2f986dcd49cd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c6f1dec453734393a99fd075094ebde7', 'request_id': 'req-c1382f2f-2a66-4b60-8a9b-2f986dcd49cd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:15.587189 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-c1382f2f-2a66-4b60-8a9b-2f986dcd49cd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c6f1dec453734393a99fd075094ebde7', 'request_id': 'req-c1382f2f-2a66-4b60-8a9b-2f986dcd49cd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:15.595525 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:15.600087 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-c1382f2f-2a66-4b60-8a9b-2f986dcd49cd admin admin] The token cache is being invalidate because project c6f1dec453734393a99fd075094ebde7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:06:15.600380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-c1382f2f-2a66-4b60-8a9b-2f986dcd49cd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:15.601623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-c1382f2f-2a66-4b60-8a9b-2f986dcd49cd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:15.603476 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1001/2014] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:06:15 2022] DELETE /identity/v3/projects/c6f1dec453734393a99fd075094ebde7 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:15.608449 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d890869b-0bce-4126-b4b5-2d4b30264efe None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:15.608990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d890869b-0bce-4126-b4b5-2d4b30264efe None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:15.608990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d890869b-0bce-4126-b4b5-2d4b30264efe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:15.629828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-d890869b-0bce-4126-b4b5-2d4b30264efe None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:06:15.631040 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-d890869b-0bce-4126-b4b5-2d4b30264efe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:15.690979 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1014/2015] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:06:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:06:15.757130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d9a108-f4e4-4841-9378-856485a43892 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:15.757368 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d9a108-f4e4-4841-9378-856485a43892 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:15.757663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d9a108-f4e4-4841-9378-856485a43892 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:15.779858 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-a3d9a108-f4e4-4841-9378-856485a43892 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:06:15.780987 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a3d9a108-f4e4-4841-9378-856485a43892 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:15.789881 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1002/2016] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:06:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:06:15.794320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d890869b-0bce-4126-b4b5-2d4b30264efe None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:15.805186 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d890869b-0bce-4126-b4b5-2d4b30264efe None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:15.811517 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fc0bd1-3a87-4e85-aed9-ef99db0cb216 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:15.812152 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fc0bd1-3a87-4e85-aed9-ef99db0cb216 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:15.812429 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fc0bd1-3a87-4e85-aed9-ef99db0cb216 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:15.812694 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fc0bd1-3a87-4e85-aed9-ef99db0cb216 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:15.813333 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fc0bd1-3a87-4e85-aed9-ef99db0cb216 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:06:15.813635 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fc0bd1-3a87-4e85-aed9-ef99db0cb216 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:15.815234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fc0bd1-3a87-4e85-aed9-ef99db0cb216 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:15.827555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1015/2017] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:06:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 01:06:15.831975 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d9a108-f4e4-4841-9378-856485a43892 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:15.842173 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d9a108-f4e4-4841-9378-856485a43892 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:15.847886 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64c5c5b-b913-4241-806b-a956d7465312 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:15.848365 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64c5c5b-b913-4241-806b-a956d7465312 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:15.848603 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64c5c5b-b913-4241-806b-a956d7465312 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:15.848860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64c5c5b-b913-4241-806b-a956d7465312 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:15.849484 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64c5c5b-b913-4241-806b-a956d7465312 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:15.851433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64c5c5b-b913-4241-806b-a956d7465312 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:15.890726 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1003/2018] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:06:15 2022] POST /identity/v3/projects => generated 385 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:06:15.891936 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fc0bd1-3a87-4e85-aed9-ef99db0cb216 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:15.902506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fc0bd1-3a87-4e85-aed9-ef99db0cb216 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:15.916008 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:15.964478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b087e84-824c-42fe-a34c-dc5ffd0e05a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:15.965266 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b087e84-824c-42fe-a34c-dc5ffd0e05a4 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:15.965531 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b087e84-824c-42fe-a34c-dc5ffd0e05a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:15.965803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b087e84-824c-42fe-a34c-dc5ffd0e05a4 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:15.966424 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b087e84-824c-42fe-a34c-dc5ffd0e05a4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:15.968356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b087e84-824c-42fe-a34c-dc5ffd0e05a4 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:15.997538 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1016/2019] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:06:15 2022] POST /identity/v3/users => generated 293 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:06:16.001122 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64c5c5b-b913-4241-806b-a956d7465312 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:16.012371 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64c5c5b-b913-4241-806b-a956d7465312 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:16.018002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4febe81-3f86-43cf-a861-aa34f6d4ae66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:16.018737 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4febe81-3f86-43cf-a861-aa34f6d4ae66 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:16.019015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4febe81-3f86-43cf-a861-aa34f6d4ae66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:16.019281 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4febe81-3f86-43cf-a861-aa34f6d4ae66 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:16.019961 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4febe81-3f86-43cf-a861-aa34f6d4ae66 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:16.021526 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4febe81-3f86-43cf-a861-aa34f6d4ae66 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:16.033270 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1004/2020] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:06:16 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:06:16.037414 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b087e84-824c-42fe-a34c-dc5ffd0e05a4 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:16.047970 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b087e84-824c-42fe-a34c-dc5ffd0e05a4 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:16.052688 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957d5179-f763-403c-9369-98dd5f2fe858 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:16.053284 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957d5179-f763-403c-9369-98dd5f2fe858 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:16.053555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957d5179-f763-403c-9369-98dd5f2fe858 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:16.053823 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957d5179-f763-403c-9369-98dd5f2fe858 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:16.054450 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957d5179-f763-403c-9369-98dd5f2fe858 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:16.056105 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957d5179-f763-403c-9369-98dd5f2fe858 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:16.068445 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1017/2021] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:06:16 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:06:16.072802 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4febe81-3f86-43cf-a861-aa34f6d4ae66 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:16.082831 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4febe81-3f86-43cf-a861-aa34f6d4ae66 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:16.088949 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43517de-241b-4eb3-b95b-ab30e5acde90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:16.090046 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43517de-241b-4eb3-b95b-ab30e5acde90 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:16.090320 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43517de-241b-4eb3-b95b-ab30e5acde90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:16.090578 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43517de-241b-4eb3-b95b-ab30e5acde90 admin admin] PATH_INFO: `/v3/projects/ff4e11982aaa4d2c9762cda693540dce/users/725fcc24201e442bb0be57eeb4700c62/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:16.106303 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43517de-241b-4eb3-b95b-ab30e5acde90 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff4e11982aaa4d2c9762cda693540dce, user_id=725fcc24201e442bb0be57eeb4700c62, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:16.109261 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43517de-241b-4eb3-b95b-ab30e5acde90 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:16.124418 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1005/2022] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:06:16 2022] PUT /identity/v3/projects/ff4e11982aaa4d2c9762cda693540dce/users/725fcc24201e442bb0be57eeb4700c62/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:16.134900 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e495c69-e739-496d-8a8e-724447de6372 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:16.135302 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e495c69-e739-496d-8a8e-724447de6372 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:16.135615 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e495c69-e739-496d-8a8e-724447de6372 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:16.163747 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-4e495c69-e739-496d-8a8e-724447de6372 None None] MFA Rules not processed for user `725fcc24201e442bb0be57eeb4700c62`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:06:16.164994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-4e495c69-e739-496d-8a8e-724447de6372 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:16.231654 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1018/2023] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:06:16 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:06:16.236425 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43517de-241b-4eb3-b95b-ab30e5acde90 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:16.301803 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43517de-241b-4eb3-b95b-ab30e5acde90 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:16.306645 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1710b2-0305-426b-992a-174822adf51b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:16.307255 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1710b2-0305-426b-992a-174822adf51b admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:16.307554 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1710b2-0305-426b-992a-174822adf51b admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:16.307971 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1710b2-0305-426b-992a-174822adf51b admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:16.308605 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1710b2-0305-426b-992a-174822adf51b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:16.310541 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1710b2-0305-426b-992a-174822adf51b admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:16.339504 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1006/2024] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:06:16 2022] POST /identity/v3/projects => generated 385 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:06:16.347527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e495c69-e739-496d-8a8e-724447de6372 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:16.358845 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e495c69-e739-496d-8a8e-724447de6372 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:16.415474 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37f67b8-2b51-4a3b-afc5-f8f6f7eca767 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:16.416244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37f67b8-2b51-4a3b-afc5-f8f6f7eca767 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:16.416523 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37f67b8-2b51-4a3b-afc5-f8f6f7eca767 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:16.416789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37f67b8-2b51-4a3b-afc5-f8f6f7eca767 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:16.417516 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37f67b8-2b51-4a3b-afc5-f8f6f7eca767 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:16.419442 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37f67b8-2b51-4a3b-afc5-f8f6f7eca767 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:16.443813 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1019/2025] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:06:16 2022] POST /identity/v3/users => generated 299 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:06:16.448077 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1710b2-0305-426b-992a-174822adf51b admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:16.459547 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1710b2-0305-426b-992a-174822adf51b admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:16.465400 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252f2bd2-bbd9-4aa0-85ca-c5caf15843c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:16.466444 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252f2bd2-bbd9-4aa0-85ca-c5caf15843c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:16.466444 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252f2bd2-bbd9-4aa0-85ca-c5caf15843c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:16.466668 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252f2bd2-bbd9-4aa0-85ca-c5caf15843c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:16.467168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-252f2bd2-bbd9-4aa0-85ca-c5caf15843c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:16.468788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-252f2bd2-bbd9-4aa0-85ca-c5caf15843c5 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:16.480256 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1007/2026] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:06:16 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:06:16.484891 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37f67b8-2b51-4a3b-afc5-f8f6f7eca767 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:16.495853 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37f67b8-2b51-4a3b-afc5-f8f6f7eca767 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:16.500520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9287b7-04a7-4b58-a04c-6bbd4b4633b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:16.501180 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9287b7-04a7-4b58-a04c-6bbd4b4633b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:16.501472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9287b7-04a7-4b58-a04c-6bbd4b4633b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:16.501748 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9287b7-04a7-4b58-a04c-6bbd4b4633b8 admin admin] PATH_INFO: `/v3/projects/1fad546675124fd89e737f8032e5ff38/users/c6d5338dc0ce40b4a0ced22736a6e8ba/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:16.514864 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:16.518465 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9287b7-04a7-4b58-a04c-6bbd4b4633b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1fad546675124fd89e737f8032e5ff38, user_id=c6d5338dc0ce40b4a0ced22736a6e8ba, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:16.521370 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9287b7-04a7-4b58-a04c-6bbd4b4633b8 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:16.538622 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1020/2027] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:06:16 2022] PUT /identity/v3/projects/1fad546675124fd89e737f8032e5ff38/users/c6d5338dc0ce40b4a0ced22736a6e8ba/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:16.542644 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252f2bd2-bbd9-4aa0-85ca-c5caf15843c5 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:16.553844 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252f2bd2-bbd9-4aa0-85ca-c5caf15843c5 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:16.609712 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d04c00-bf35-48db-9703-18d06bf4fe0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:16.610328 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d04c00-bf35-48db-9703-18d06bf4fe0c admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:16.610601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d04c00-bf35-48db-9703-18d06bf4fe0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:16.610869 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d04c00-bf35-48db-9703-18d06bf4fe0c admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:16.611506 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d04c00-bf35-48db-9703-18d06bf4fe0c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:16.613146 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d04c00-bf35-48db-9703-18d06bf4fe0c admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:16.626811 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1008/2028] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:06:16 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:06:16.628500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9287b7-04a7-4b58-a04c-6bbd4b4633b8 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:16.639534 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9287b7-04a7-4b58-a04c-6bbd4b4633b8 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:16.644155 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73faf19f-6c76-40c3-ae5d-f40e0cc58cbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:16.644876 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73faf19f-6c76-40c3-ae5d-f40e0cc58cbc admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:16.645266 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73faf19f-6c76-40c3-ae5d-f40e0cc58cbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:16.645559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73faf19f-6c76-40c3-ae5d-f40e0cc58cbc admin admin] PATH_INFO: `/v3/projects/1fad546675124fd89e737f8032e5ff38/users/c6d5338dc0ce40b4a0ced22736a6e8ba/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:16.647425 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73faf19f-6c76-40c3-ae5d-f40e0cc58cbc admin admin] RBAC: Authorizing `identity:create_grant(project_id=1fad546675124fd89e737f8032e5ff38, user_id=c6d5338dc0ce40b4a0ced22736a6e8ba, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:16.650080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73faf19f-6c76-40c3-ae5d-f40e0cc58cbc admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:16.661196 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1021/2029] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:06:16 2022] PUT /identity/v3/projects/1fad546675124fd89e737f8032e5ff38/users/c6d5338dc0ce40b4a0ced22736a6e8ba/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:16.672210 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaac9021-f62a-4d91-a42e-134520494592 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:16.672210 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaac9021-f62a-4d91-a42e-134520494592 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:16.672476 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaac9021-f62a-4d91-a42e-134520494592 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:16.687912 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:16.703148 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-aaac9021-f62a-4d91-a42e-134520494592 None None] MFA Rules not processed for user `c6d5338dc0ce40b4a0ced22736a6e8ba`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:06:16.704423 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-aaac9021-f62a-4d91-a42e-134520494592 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:16.775506 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1009/2030] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:06:16 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:06:16.786843 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73faf19f-6c76-40c3-ae5d-f40e0cc58cbc admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:16.788762 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-73faf19f-6c76-40c3-ae5d-f40e0cc58cbc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:16.848613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73faf19f-6c76-40c3-ae5d-f40e0cc58cbc admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:16.852383 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c0c528-bf23-44f4-9cca-0d205e2502aa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:16.852988 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c0c528-bf23-44f4-9cca-0d205e2502aa service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:16.853271 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c0c528-bf23-44f4-9cca-0d205e2502aa service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:16.853537 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c0c528-bf23-44f4-9cca-0d205e2502aa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:16.863206 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c0c528-bf23-44f4-9cca-0d205e2502aa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:16.864785 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c0c528-bf23-44f4-9cca-0d205e2502aa service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:16.874740 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1022/2031] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:06:16 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:06:17.519585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaac9021-f62a-4d91-a42e-134520494592 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:17.585465 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaac9021-f62a-4d91-a42e-134520494592 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:17.590847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdc2b11-8bf1-4232-a3a3-d81324dd6d0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:17.591587 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdc2b11-8bf1-4232-a3a3-d81324dd6d0d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:17.591926 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdc2b11-8bf1-4232-a3a3-d81324dd6d0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:17.592176 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdc2b11-8bf1-4232-a3a3-d81324dd6d0d admin admin] PATH_INFO: `/v3/users/725fcc24201e442bb0be57eeb4700c62` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:17.593442 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdc2b11-8bf1-4232-a3a3-d81324dd6d0d admin admin] RBAC: Authorizing `identity:delete_user(user_id=725fcc24201e442bb0be57eeb4700c62)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:17.595376 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdc2b11-8bf1-4232-a3a3-d81324dd6d0d admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:17.676063 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-9bdc2b11-8bf1-4232-a3a3-d81324dd6d0d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '725fcc24201e442bb0be57eeb4700c62'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:17.692438 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-9bdc2b11-8bf1-4232-a3a3-d81324dd6d0d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '725fcc24201e442bb0be57eeb4700c62'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:17.693184 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-9bdc2b11-8bf1-4232-a3a3-d81324dd6d0d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '725fcc24201e442bb0be57eeb4700c62'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:17.704547 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-9bdc2b11-8bf1-4232-a3a3-d81324dd6d0d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '725fcc24201e442bb0be57eeb4700c62'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:17.727419 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-9bdc2b11-8bf1-4232-a3a3-d81324dd6d0d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '725fcc24201e442bb0be57eeb4700c62'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:17.735543 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1010/2032] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:06:17 2022] DELETE /identity/v3/users/725fcc24201e442bb0be57eeb4700c62 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:17.740134 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e2b896-c16b-4a7f-bf16-c820c6848afb None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:17.740628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e2b896-c16b-4a7f-bf16-c820c6848afb None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:17.740727 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e2b896-c16b-4a7f-bf16-c820c6848afb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:17.761385 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-32e2b896-c16b-4a7f-bf16-c820c6848afb None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:06:17.762660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-32e2b896-c16b-4a7f-bf16-c820c6848afb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:17.841448 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1023/2033] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:06:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:06:17.851436 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdc2b11-8bf1-4232-a3a3-d81324dd6d0d admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:17.852952 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-9bdc2b11-8bf1-4232-a3a3-d81324dd6d0d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:17.867326 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:17.915216 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdc2b11-8bf1-4232-a3a3-d81324dd6d0d admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:17.919614 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131ad693-6c4b-4795-b7d3-c747158b8feb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:17.920261 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131ad693-6c4b-4795-b7d3-c747158b8feb service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:17.920535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131ad693-6c4b-4795-b7d3-c747158b8feb service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:17.920803 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131ad693-6c4b-4795-b7d3-c747158b8feb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:17.931296 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131ad693-6c4b-4795-b7d3-c747158b8feb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:17.932896 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131ad693-6c4b-4795-b7d3-c747158b8feb service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:17.943193 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1011/2034] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:06:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:06:18.221607 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e2b896-c16b-4a7f-bf16-c820c6848afb None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:18.223531 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-32e2b896-c16b-4a7f-bf16-c820c6848afb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:18.237995 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e2b896-c16b-4a7f-bf16-c820c6848afb None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:18.243471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7370ad2-bf72-41a8-9abe-cdbbffc8014b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:18.244217 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7370ad2-bf72-41a8-9abe-cdbbffc8014b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:18.244493 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7370ad2-bf72-41a8-9abe-cdbbffc8014b admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:18.244863 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7370ad2-bf72-41a8-9abe-cdbbffc8014b admin admin] PATH_INFO: `/v3/projects/ff4e11982aaa4d2c9762cda693540dce` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:18.246038 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7370ad2-bf72-41a8-9abe-cdbbffc8014b admin admin] RBAC: Authorizing `identity:delete_project(project_id=ff4e11982aaa4d2c9762cda693540dce)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:18.248080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7370ad2-bf72-41a8-9abe-cdbbffc8014b admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:18.294931 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-e7370ad2-bf72-41a8-9abe-cdbbffc8014b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ff4e11982aaa4d2c9762cda693540dce', 'request_id': 'req-e7370ad2-bf72-41a8-9abe-cdbbffc8014b'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:18.302584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-e7370ad2-bf72-41a8-9abe-cdbbffc8014b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ff4e11982aaa4d2c9762cda693540dce', 'request_id': 'req-e7370ad2-bf72-41a8-9abe-cdbbffc8014b'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:18.309299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-e7370ad2-bf72-41a8-9abe-cdbbffc8014b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ff4e11982aaa4d2c9762cda693540dce', 'request_id': 'req-e7370ad2-bf72-41a8-9abe-cdbbffc8014b'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:18.321665 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-e7370ad2-bf72-41a8-9abe-cdbbffc8014b admin admin] The token cache is being invalidate because project ff4e11982aaa4d2c9762cda693540dce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:06:18.321946 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-e7370ad2-bf72-41a8-9abe-cdbbffc8014b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:18.322600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-e7370ad2-bf72-41a8-9abe-cdbbffc8014b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:18.324363 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1024/2035] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:06:18 2022] DELETE /identity/v3/projects/ff4e11982aaa4d2c9762cda693540dce => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:18.328699 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131ad693-6c4b-4795-b7d3-c747158b8feb service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:18.338848 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131ad693-6c4b-4795-b7d3-c747158b8feb service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:18.397652 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5586e4a0-70d6-4185-a7d7-78ec98991ead admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:18.398288 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5586e4a0-70d6-4185-a7d7-78ec98991ead admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:18.398567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5586e4a0-70d6-4185-a7d7-78ec98991ead admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:18.398869 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5586e4a0-70d6-4185-a7d7-78ec98991ead admin admin] PATH_INFO: `/v3/users/c6d5338dc0ce40b4a0ced22736a6e8ba` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:18.400062 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5586e4a0-70d6-4185-a7d7-78ec98991ead admin admin] RBAC: Authorizing `identity:delete_user(user_id=c6d5338dc0ce40b4a0ced22736a6e8ba)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:18.402118 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5586e4a0-70d6-4185-a7d7-78ec98991ead admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:18.453096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-5586e4a0-70d6-4185-a7d7-78ec98991ead admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c6d5338dc0ce40b4a0ced22736a6e8ba'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:18.460906 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-5586e4a0-70d6-4185-a7d7-78ec98991ead admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c6d5338dc0ce40b4a0ced22736a6e8ba'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:18.461796 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-5586e4a0-70d6-4185-a7d7-78ec98991ead admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c6d5338dc0ce40b4a0ced22736a6e8ba'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:18.469291 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-5586e4a0-70d6-4185-a7d7-78ec98991ead admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c6d5338dc0ce40b4a0ced22736a6e8ba'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:18.485846 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-5586e4a0-70d6-4185-a7d7-78ec98991ead admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c6d5338dc0ce40b4a0ced22736a6e8ba'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:18.492637 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1012/2036] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:06:18 2022] DELETE /identity/v3/users/c6d5338dc0ce40b4a0ced22736a6e8ba => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:18.780832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7370ad2-bf72-41a8-9abe-cdbbffc8014b admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:18.782807 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e7370ad2-bf72-41a8-9abe-cdbbffc8014b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:18.793322 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:18.798420 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7370ad2-bf72-41a8-9abe-cdbbffc8014b admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:18.833409 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72f352c-01d3-4bdf-97dc-3d236b820bcb None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:18.833903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72f352c-01d3-4bdf-97dc-3d236b820bcb None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:18.834051 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72f352c-01d3-4bdf-97dc-3d236b820bcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:18.875216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937f9445-6cec-47a3-8683-a4bd77ad78da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:18.875895 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937f9445-6cec-47a3-8683-a4bd77ad78da admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:18.876422 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937f9445-6cec-47a3-8683-a4bd77ad78da admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:18.876724 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937f9445-6cec-47a3-8683-a4bd77ad78da admin admin] PATH_INFO: `/v3/projects/1fad546675124fd89e737f8032e5ff38` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:18.877873 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937f9445-6cec-47a3-8683-a4bd77ad78da admin admin] RBAC: Authorizing `identity:delete_project(project_id=1fad546675124fd89e737f8032e5ff38)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:18.879827 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937f9445-6cec-47a3-8683-a4bd77ad78da admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:18.885871 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:18.904599 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-e72f352c-01d3-4bdf-97dc-3d236b820bcb None None] MFA Rules not processed for user `4716e7afe2784e2a9b872996d895c374`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:06:18.906024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-e72f352c-01d3-4bdf-97dc-3d236b820bcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:18.938718 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-937f9445-6cec-47a3-8683-a4bd77ad78da admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1fad546675124fd89e737f8032e5ff38', 'request_id': 'req-937f9445-6cec-47a3-8683-a4bd77ad78da'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:18.947002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-937f9445-6cec-47a3-8683-a4bd77ad78da admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1fad546675124fd89e737f8032e5ff38', 'request_id': 'req-937f9445-6cec-47a3-8683-a4bd77ad78da'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:18.955197 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-937f9445-6cec-47a3-8683-a4bd77ad78da admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1fad546675124fd89e737f8032e5ff38', 'request_id': 'req-937f9445-6cec-47a3-8683-a4bd77ad78da'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:18.970816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-937f9445-6cec-47a3-8683-a4bd77ad78da admin admin] The token cache is being invalidate because project 1fad546675124fd89e737f8032e5ff38 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:06:18.971016 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-937f9445-6cec-47a3-8683-a4bd77ad78da admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:18.971756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-937f9445-6cec-47a3-8683-a4bd77ad78da admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:06:18.975797 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1025/2037] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:06:18 2022] DELETE /identity/v3/projects/1fad546675124fd89e737f8032e5ff38 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:18.981257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1013/2038] 10.222.0.16 () {58 vars in 1009 bytes} [Sun Oct 30 01:06:18 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:06:18.984189 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b685879-c262-444d-b805-43f5b389d9ea None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:18.984475 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b685879-c262-444d-b805-43f5b389d9ea None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:18.984747 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b685879-c262-444d-b805-43f5b389d9ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:19.010973 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-4b685879-c262-444d-b805-43f5b389d9ea None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:06:19.012296 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-4b685879-c262-444d-b805-43f5b389d9ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:19.032312 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72f352c-01d3-4bdf-97dc-3d236b820bcb None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:19.033721 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-e72f352c-01d3-4bdf-97dc-3d236b820bcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:19.072488 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1026/2039] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:06:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:06:19.080673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1401ad5-bb8f-4add-9b9f-1e7871afaece None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:19.080980 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1401ad5-bb8f-4add-9b9f-1e7871afaece None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:19.081234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1401ad5-bb8f-4add-9b9f-1e7871afaece None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:19.102741 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-e1401ad5-bb8f-4add-9b9f-1e7871afaece None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:06:19.103743 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e1401ad5-bb8f-4add-9b9f-1e7871afaece None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:19.112295 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1027/2040] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:06:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:06:19.116647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1401ad5-bb8f-4add-9b9f-1e7871afaece None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:19.125782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1401ad5-bb8f-4add-9b9f-1e7871afaece None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:19.129679 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72f352c-01d3-4bdf-97dc-3d236b820bcb None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:19.130405 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032a064a-8a88-4969-8692-01dbf276bb43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:19.130998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032a064a-8a88-4969-8692-01dbf276bb43 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:19.131270 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032a064a-8a88-4969-8692-01dbf276bb43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:19.131527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032a064a-8a88-4969-8692-01dbf276bb43 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:19.132211 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032a064a-8a88-4969-8692-01dbf276bb43 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:06:19.132506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032a064a-8a88-4969-8692-01dbf276bb43 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:19.134086 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032a064a-8a88-4969-8692-01dbf276bb43 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:19.145937 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1028/2041] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:06:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 01:06:19.150365 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032a064a-8a88-4969-8692-01dbf276bb43 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:19.160054 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032a064a-8a88-4969-8692-01dbf276bb43 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:19.164746 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f256bd-8e5b-4b6d-96dc-ad5716bab5a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:19.165332 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f256bd-8e5b-4b6d-96dc-ad5716bab5a1 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:19.165606 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f256bd-8e5b-4b6d-96dc-ad5716bab5a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:19.165875 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f256bd-8e5b-4b6d-96dc-ad5716bab5a1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:19.166498 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f256bd-8e5b-4b6d-96dc-ad5716bab5a1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:19.168468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f256bd-8e5b-4b6d-96dc-ad5716bab5a1 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:19.184096 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:19.193070 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0469f9-9df7-473f-814a-f5abb615a09c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '67f6eff8c0254cc6a210101dd7af988d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:19.193708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0469f9-9df7-473f-814a-f5abb615a09c service swift] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:19.193983 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0469f9-9df7-473f-814a-f5abb615a09c service swift] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:19.194237 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0469f9-9df7-473f-814a-f5abb615a09c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:19.196170 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-ce0469f9-9df7-473f-814a-f5abb615a09c service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:19.203791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1029/2042] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:06:19 2022] POST /identity/v3/projects => generated 369 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:06:19.204758 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f256bd-8e5b-4b6d-96dc-ad5716bab5a1 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:19.217190 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:19.256588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0469f9-9df7-473f-814a-f5abb615a09c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:19.258182 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0469f9-9df7-473f-814a-f5abb615a09c service swift] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:19.266325 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1014/2043] 10.222.0.16 () {60 vars in 1384 bytes} [Sun Oct 30 01:06:19 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 30 01:06:19.269108 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f256bd-8e5b-4b6d-96dc-ad5716bab5a1 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:19.277175 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18aaaa5d-8cc9-4126-a50a-3040335c052d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:19.278239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18aaaa5d-8cc9-4126-a50a-3040335c052d admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:19.278239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18aaaa5d-8cc9-4126-a50a-3040335c052d admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:19.278487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18aaaa5d-8cc9-4126-a50a-3040335c052d admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:19.279032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18aaaa5d-8cc9-4126-a50a-3040335c052d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:19.281106 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18aaaa5d-8cc9-4126-a50a-3040335c052d admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:19.308982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1030/2044] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:06:19 2022] POST /identity/v3/users => generated 285 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:06:19.310425 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0469f9-9df7-473f-814a-f5abb615a09c service swift] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:19.321174 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0469f9-9df7-473f-814a-f5abb615a09c service swift] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:19.326553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58d2d7f-565f-426c-b2cb-bd0c30fd331f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:19.327225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58d2d7f-565f-426c-b2cb-bd0c30fd331f admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:19.327629 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58d2d7f-565f-426c-b2cb-bd0c30fd331f admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:19.327941 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58d2d7f-565f-426c-b2cb-bd0c30fd331f admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:19.328482 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58d2d7f-565f-426c-b2cb-bd0c30fd331f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:19.330102 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58d2d7f-565f-426c-b2cb-bd0c30fd331f admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:19.346323 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1015/2045] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:06:19 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:06:19.347779 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18aaaa5d-8cc9-4126-a50a-3040335c052d admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:19.360974 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18aaaa5d-8cc9-4126-a50a-3040335c052d admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:19.366717 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5b42b2-f500-46c8-832c-35d066c4d151 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:19.367199 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5b42b2-f500-46c8-832c-35d066c4d151 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:19.369762 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5b42b2-f500-46c8-832c-35d066c4d151 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:19.370185 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5b42b2-f500-46c8-832c-35d066c4d151 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:19.371110 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5b42b2-f500-46c8-832c-35d066c4d151 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:19.372950 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5b42b2-f500-46c8-832c-35d066c4d151 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:19.384502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1031/2046] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:06:19 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:06:19.389295 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58d2d7f-565f-426c-b2cb-bd0c30fd331f admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:19.399992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58d2d7f-565f-426c-b2cb-bd0c30fd331f admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:19.404931 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aacbb397-36e3-40db-bfd9-2f9e1fe4e6d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:19.405594 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacbb397-36e3-40db-bfd9-2f9e1fe4e6d4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:19.405866 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacbb397-36e3-40db-bfd9-2f9e1fe4e6d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:19.406130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacbb397-36e3-40db-bfd9-2f9e1fe4e6d4 admin admin] PATH_INFO: `/v3/projects/9f38d8fafa38475380256af363f7f81b/users/cf5437668fee49938364a2fdf8a02538/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:19.423204 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aacbb397-36e3-40db-bfd9-2f9e1fe4e6d4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f38d8fafa38475380256af363f7f81b, user_id=cf5437668fee49938364a2fdf8a02538, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:19.426239 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aacbb397-36e3-40db-bfd9-2f9e1fe4e6d4 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:19.450830 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1016/2047] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:06:19 2022] PUT /identity/v3/projects/9f38d8fafa38475380256af363f7f81b/users/cf5437668fee49938364a2fdf8a02538/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:06:19.452838 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b14a2c9-8b03-481a-8f73-2b290b636341 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:19.453127 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b14a2c9-8b03-481a-8f73-2b290b636341 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:19.453396 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b14a2c9-8b03-481a-8f73-2b290b636341 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:19.485858 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-2b14a2c9-8b03-481a-8f73-2b290b636341 None None] MFA Rules not processed for user `cf5437668fee49938364a2fdf8a02538`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:06:19.487072 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-2b14a2c9-8b03-481a-8f73-2b290b636341 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:19.560527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1032/2048] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:06:19 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:06:19.574657 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aacbb397-36e3-40db-bfd9-2f9e1fe4e6d4 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:19.576856 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-aacbb397-36e3-40db-bfd9-2f9e1fe4e6d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:19.645754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aacbb397-36e3-40db-bfd9-2f9e1fe4e6d4 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:19.650474 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34590e0f-dcb1-4dfb-874f-bc78dcfd4416 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '807b123652d245269336fcffaf36478e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:19.651175 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34590e0f-dcb1-4dfb-874f-bc78dcfd4416 service glance] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:19.651597 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34590e0f-dcb1-4dfb-874f-bc78dcfd4416 service glance] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:19.651969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34590e0f-dcb1-4dfb-874f-bc78dcfd4416 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:19.679052 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34590e0f-dcb1-4dfb-874f-bc78dcfd4416 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:19.679052 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34590e0f-dcb1-4dfb-874f-bc78dcfd4416 service glance] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:19.703557 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1017/2049] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:06:19 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:06:29.552916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:INSERT=4,keystone:SELECT=68 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:29.693654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=59,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:39.332251 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b14a2c9-8b03-481a-8f73-2b290b636341 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:39.342177 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b14a2c9-8b03-481a-8f73-2b290b636341 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:39.345828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d89195-681e-4cce-ad79-25df9503947d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '807b123652d245269336fcffaf36478e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:39.346429 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d89195-681e-4cce-ad79-25df9503947d service glance] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:39.346861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d89195-681e-4cce-ad79-25df9503947d service glance] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:39.347128 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d89195-681e-4cce-ad79-25df9503947d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:39.349065 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-55d89195-681e-4cce-ad79-25df9503947d service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:39.430265 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d89195-681e-4cce-ad79-25df9503947d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:39.431851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d89195-681e-4cce-ad79-25df9503947d service glance] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:39.445879 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1033/2050] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:06:39 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:06:39.518101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34590e0f-dcb1-4dfb-874f-bc78dcfd4416 service glance] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:39.519917 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-34590e0f-dcb1-4dfb-874f-bc78dcfd4416 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:39.579189 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34590e0f-dcb1-4dfb-874f-bc78dcfd4416 service glance] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:39.582877 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a60559-d7a0-4d30-8d71-4852878b31f7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:39.583502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a60559-d7a0-4d30-8d71-4852878b31f7 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:39.583771 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a60559-d7a0-4d30-8d71-4852878b31f7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:39.584059 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a60559-d7a0-4d30-8d71-4852878b31f7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:39.585948 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-24a60559-d7a0-4d30-8d71-4852878b31f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:39.596111 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a60559-d7a0-4d30-8d71-4852878b31f7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:39.597522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a60559-d7a0-4d30-8d71-4852878b31f7 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:39.605674 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1018/2051] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:06:39 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:06:39.680573 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d89195-681e-4cce-ad79-25df9503947d service glance] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:39.690070 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d89195-681e-4cce-ad79-25df9503947d service glance] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:39.693605 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e174cdd3-33a4-40f6-acd9-6eeabba08e89 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:39.694220 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e174cdd3-33a4-40f6-acd9-6eeabba08e89 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:39.694478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e174cdd3-33a4-40f6-acd9-6eeabba08e89 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:39.694747 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e174cdd3-33a4-40f6-acd9-6eeabba08e89 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:39.696471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e174cdd3-33a4-40f6-acd9-6eeabba08e89 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:39.709252 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e174cdd3-33a4-40f6-acd9-6eeabba08e89 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:39.710742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e174cdd3-33a4-40f6-acd9-6eeabba08e89 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:39.721490 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1034/2052] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:06:39 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:06:39.786711 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a60559-d7a0-4d30-8d71-4852878b31f7 service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:39.788235 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-24a60559-d7a0-4d30-8d71-4852878b31f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:39.858496 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a60559-d7a0-4d30-8d71-4852878b31f7 service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:39.863934 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d8836c-a46d-412a-96f5-6af163e26509 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '63cc035057064805a95e9f12714ebade', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:39.864586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d8836c-a46d-412a-96f5-6af163e26509 service placement] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:39.864853 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d8836c-a46d-412a-96f5-6af163e26509 service placement] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:39.865114 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d8836c-a46d-412a-96f5-6af163e26509 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:39.867140 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-39d8836c-a46d-412a-96f5-6af163e26509 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:39.878342 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d8836c-a46d-412a-96f5-6af163e26509 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:39.879940 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d8836c-a46d-412a-96f5-6af163e26509 service placement] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:39.889580 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1019/2053] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:06:39 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:06:42.439902 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66aa4690-9c58-4986-bc15-4454489e3c9a None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:42.440405 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66aa4690-9c58-4986-bc15-4454489e3c9a None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:42.440496 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66aa4690-9c58-4986-bc15-4454489e3c9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:42.460290 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-66aa4690-9c58-4986-bc15-4454489e3c9a None None] MFA Rules not processed for user `4716e7afe2784e2a9b872996d895c374`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:06:42.461570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-66aa4690-9c58-4986-bc15-4454489e3c9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:42.518729 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1035/2054] 10.222.0.16 () {58 vars in 1009 bytes} [Sun Oct 30 01:06:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:06:42.531505 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d8836c-a46d-412a-96f5-6af163e26509 service placement] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:42.532961 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-39d8836c-a46d-412a-96f5-6af163e26509 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:42.632801 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d8836c-a46d-412a-96f5-6af163e26509 service placement] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:42.647105 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd41959-58e4-4d8c-9cd6-62dafe37d1e6 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '67f6eff8c0254cc6a210101dd7af988d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:42.647855 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd41959-58e4-4d8c-9cd6-62dafe37d1e6 service swift] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:42.648132 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd41959-58e4-4d8c-9cd6-62dafe37d1e6 service swift] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:42.648398 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd41959-58e4-4d8c-9cd6-62dafe37d1e6 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:42.658435 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd41959-58e4-4d8c-9cd6-62dafe37d1e6 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:42.660056 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd41959-58e4-4d8c-9cd6-62dafe37d1e6 service swift] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:42.670586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1020/2055] 10.222.0.16 () {60 vars in 1384 bytes} [Sun Oct 30 01:06:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 30 01:06:48.256891 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66aa4690-9c58-4986-bc15-4454489e3c9a None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:06:48.258328 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-66aa4690-9c58-4986-bc15-4454489e3c9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:48.274507 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66aa4690-9c58-4986-bc15-4454489e3c9a None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:06:48.279927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa66636-261f-4311-bbae-7940b5964529 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:06:48.280537 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa66636-261f-4311-bbae-7940b5964529 service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:06:48.280808 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa66636-261f-4311-bbae-7940b5964529 service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:06:48.281073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa66636-261f-4311-bbae-7940b5964529 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:06:48.282758 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-0aa66636-261f-4311-bbae-7940b5964529 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:06:48.370707 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa66636-261f-4311-bbae-7940b5964529 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:06:48.372573 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa66636-261f-4311-bbae-7940b5964529 service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:06:48.376590 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:48.395147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1036/2056] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:06:48 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:06:52.666896 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=90 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:06:58.392166 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=6 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:07:06.009764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd41959-58e4-4d8c-9cd6-62dafe37d1e6 service swift] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:06.074489 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd41959-58e4-4d8c-9cd6-62dafe37d1e6 service swift] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:06.079610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499b0d93-fe8c-4f47-a614-fda773aea778 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:06.080380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499b0d93-fe8c-4f47-a614-fda773aea778 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:06.083113 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499b0d93-fe8c-4f47-a614-fda773aea778 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:06.083113 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499b0d93-fe8c-4f47-a614-fda773aea778 admin admin] PATH_INFO: `/v3/users/cf5437668fee49938364a2fdf8a02538` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:06.083113 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499b0d93-fe8c-4f47-a614-fda773aea778 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cf5437668fee49938364a2fdf8a02538)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:06.084516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499b0d93-fe8c-4f47-a614-fda773aea778 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:06.140863 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-499b0d93-fe8c-4f47-a614-fda773aea778 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cf5437668fee49938364a2fdf8a02538'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:06.150905 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-499b0d93-fe8c-4f47-a614-fda773aea778 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cf5437668fee49938364a2fdf8a02538'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:06.151560 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-499b0d93-fe8c-4f47-a614-fda773aea778 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cf5437668fee49938364a2fdf8a02538'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:06.159823 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-499b0d93-fe8c-4f47-a614-fda773aea778 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cf5437668fee49938364a2fdf8a02538'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:06.173355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-499b0d93-fe8c-4f47-a614-fda773aea778 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cf5437668fee49938364a2fdf8a02538'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:06.179520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1021/2057] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:07:06 2022] DELETE /identity/v3/users/cf5437668fee49938364a2fdf8a02538 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:07:06.184318 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e00877-312c-4e04-b89c-de3f38efb4af None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:06.184904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e00877-312c-4e04-b89c-de3f38efb4af None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:06.184904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e00877-312c-4e04-b89c-de3f38efb4af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:06.206219 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-70e00877-312c-4e04-b89c-de3f38efb4af None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:07:06.207461 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-70e00877-312c-4e04-b89c-de3f38efb4af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:06.267290 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1037/2058] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:07:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:07:06.279901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499b0d93-fe8c-4f47-a614-fda773aea778 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:06.282302 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-499b0d93-fe8c-4f47-a614-fda773aea778 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:06.350494 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499b0d93-fe8c-4f47-a614-fda773aea778 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:06.354637 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f8f41e-90a6-478a-85e7-cf6fd8a55b5a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:06.355224 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f8f41e-90a6-478a-85e7-cf6fd8a55b5a service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:06.355492 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f8f41e-90a6-478a-85e7-cf6fd8a55b5a service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:06.355784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f8f41e-90a6-478a-85e7-cf6fd8a55b5a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:06.365744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f8f41e-90a6-478a-85e7-cf6fd8a55b5a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:06.367172 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f8f41e-90a6-478a-85e7-cf6fd8a55b5a service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:06.377193 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1022/2059] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:07:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:07:06.654449 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e00877-312c-4e04-b89c-de3f38efb4af None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:06.655940 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-70e00877-312c-4e04-b89c-de3f38efb4af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:06.667565 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e00877-312c-4e04-b89c-de3f38efb4af None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:06.673568 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcf811c-9663-4f1d-b1e4-d9e9da018c65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:06.674207 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcf811c-9663-4f1d-b1e4-d9e9da018c65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:06.674482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcf811c-9663-4f1d-b1e4-d9e9da018c65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:06.674751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcf811c-9663-4f1d-b1e4-d9e9da018c65 admin admin] PATH_INFO: `/v3/projects/9f38d8fafa38475380256af363f7f81b` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:06.676042 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bcf811c-9663-4f1d-b1e4-d9e9da018c65 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9f38d8fafa38475380256af363f7f81b)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:06.677812 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bcf811c-9663-4f1d-b1e4-d9e9da018c65 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:06.730332 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-5bcf811c-9663-4f1d-b1e4-d9e9da018c65 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9f38d8fafa38475380256af363f7f81b', 'request_id': 'req-5bcf811c-9663-4f1d-b1e4-d9e9da018c65'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:06.741808 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-5bcf811c-9663-4f1d-b1e4-d9e9da018c65 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9f38d8fafa38475380256af363f7f81b', 'request_id': 'req-5bcf811c-9663-4f1d-b1e4-d9e9da018c65'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:06.748147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-5bcf811c-9663-4f1d-b1e4-d9e9da018c65 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9f38d8fafa38475380256af363f7f81b', 'request_id': 'req-5bcf811c-9663-4f1d-b1e4-d9e9da018c65'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:06.764280 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-5bcf811c-9663-4f1d-b1e4-d9e9da018c65 admin admin] The token cache is being invalidate because project 9f38d8fafa38475380256af363f7f81b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:07:06.764537 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-5bcf811c-9663-4f1d-b1e4-d9e9da018c65 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:06.765310 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-5bcf811c-9663-4f1d-b1e4-d9e9da018c65 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:06.766928 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1038/2060] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:07:06 2022] DELETE /identity/v3/projects/9f38d8fafa38475380256af363f7f81b => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:07:06.772343 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694600dc-06ae-47f0-9f83-f8d12835fc80 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:06.772623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694600dc-06ae-47f0-9f83-f8d12835fc80 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:06.772887 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694600dc-06ae-47f0-9f83-f8d12835fc80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:06.798210 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-694600dc-06ae-47f0-9f83-f8d12835fc80 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:07:06.799379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-694600dc-06ae-47f0-9f83-f8d12835fc80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:06.817358 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:07:06.862325 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1023/2061] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:07:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:07:06.869823 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab34036f-2b3a-4a53-9a45-8b5f112d6b01 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:06.870096 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab34036f-2b3a-4a53-9a45-8b5f112d6b01 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:06.870361 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab34036f-2b3a-4a53-9a45-8b5f112d6b01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:06.890177 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-ab34036f-2b3a-4a53-9a45-8b5f112d6b01 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:07:06.891356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ab34036f-2b3a-4a53-9a45-8b5f112d6b01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:06.899243 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1039/2062] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:07:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:07:06.903834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694600dc-06ae-47f0-9f83-f8d12835fc80 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:06.913921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694600dc-06ae-47f0-9f83-f8d12835fc80 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:06.919654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7772496a-9a17-442a-ad2a-22101f0a1cc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:06.920298 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7772496a-9a17-442a-ad2a-22101f0a1cc2 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:06.920573 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7772496a-9a17-442a-ad2a-22101f0a1cc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:06.920888 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7772496a-9a17-442a-ad2a-22101f0a1cc2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:06.921536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7772496a-9a17-442a-ad2a-22101f0a1cc2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:07:06.921821 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7772496a-9a17-442a-ad2a-22101f0a1cc2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:06.923403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7772496a-9a17-442a-ad2a-22101f0a1cc2 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:06.934993 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1024/2063] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:07:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 01:07:06.939887 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab34036f-2b3a-4a53-9a45-8b5f112d6b01 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:06.949807 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab34036f-2b3a-4a53-9a45-8b5f112d6b01 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:06.955352 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7770a1-e15e-4684-bc6b-4be8670700df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:06.955987 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7770a1-e15e-4684-bc6b-4be8670700df admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:06.956255 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7770a1-e15e-4684-bc6b-4be8670700df admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:06.956517 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7770a1-e15e-4684-bc6b-4be8670700df admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:06.957151 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7770a1-e15e-4684-bc6b-4be8670700df admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:06.959066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7770a1-e15e-4684-bc6b-4be8670700df admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:06.990193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1040/2064] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:07:06 2022] POST /identity/v3/projects => generated 371 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:07:06.994497 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7772496a-9a17-442a-ad2a-22101f0a1cc2 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:07.004612 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7772496a-9a17-442a-ad2a-22101f0a1cc2 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:07.068976 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f060b54c-ace9-4220-bc09-4ac93e4a3826 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:07.069654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f060b54c-ace9-4220-bc09-4ac93e4a3826 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:07.069835 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f060b54c-ace9-4220-bc09-4ac93e4a3826 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:07.070126 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f060b54c-ace9-4220-bc09-4ac93e4a3826 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:07.070725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f060b54c-ace9-4220-bc09-4ac93e4a3826 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:07.072723 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f060b54c-ace9-4220-bc09-4ac93e4a3826 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:07.097205 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1025/2065] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:07:06 2022] POST /identity/v3/users => generated 286 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:07:07.101359 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7770a1-e15e-4684-bc6b-4be8670700df admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:07.113004 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7770a1-e15e-4684-bc6b-4be8670700df admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:07.117765 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6112d48d-057e-4225-86ff-2a72a41453ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:07.118399 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6112d48d-057e-4225-86ff-2a72a41453ec admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:07.118666 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6112d48d-057e-4225-86ff-2a72a41453ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:07.118927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6112d48d-057e-4225-86ff-2a72a41453ec admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:07.119567 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6112d48d-057e-4225-86ff-2a72a41453ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:07.121217 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6112d48d-057e-4225-86ff-2a72a41453ec admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:07.131477 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1041/2066] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:07:07 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:07:07.135748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f060b54c-ace9-4220-bc09-4ac93e4a3826 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:07.147025 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f060b54c-ace9-4220-bc09-4ac93e4a3826 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:07.152970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48585d4-c68c-4de7-a7ee-0aaaac15dd47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:07.153701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48585d4-c68c-4de7-a7ee-0aaaac15dd47 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:07.153981 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48585d4-c68c-4de7-a7ee-0aaaac15dd47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:07.154264 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48585d4-c68c-4de7-a7ee-0aaaac15dd47 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:07.154893 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48585d4-c68c-4de7-a7ee-0aaaac15dd47 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:07.156542 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48585d4-c68c-4de7-a7ee-0aaaac15dd47 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:07.167605 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1026/2067] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:07:07 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:07:07.171836 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6112d48d-057e-4225-86ff-2a72a41453ec admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:07.181845 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6112d48d-057e-4225-86ff-2a72a41453ec admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:07.187237 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced680c4-9867-4c53-a7d6-2f7f66f95e08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:07.187861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced680c4-9867-4c53-a7d6-2f7f66f95e08 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:07.188167 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced680c4-9867-4c53-a7d6-2f7f66f95e08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:07.188428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced680c4-9867-4c53-a7d6-2f7f66f95e08 admin admin] PATH_INFO: `/v3/projects/55f89ce52f674f84973f4ea38be311a4/users/cee0bf1da3de453fb77c7018ada3c9d8/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:07.205342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced680c4-9867-4c53-a7d6-2f7f66f95e08 admin admin] RBAC: Authorizing `identity:create_grant(project_id=55f89ce52f674f84973f4ea38be311a4, user_id=cee0bf1da3de453fb77c7018ada3c9d8, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:07.208037 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced680c4-9867-4c53-a7d6-2f7f66f95e08 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:07.224137 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1042/2068] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:07:07 2022] PUT /identity/v3/projects/55f89ce52f674f84973f4ea38be311a4/users/cee0bf1da3de453fb77c7018ada3c9d8/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:07:07.234854 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25280a1f-c107-4478-a0fa-9e72f2752719 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:07.235195 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25280a1f-c107-4478-a0fa-9e72f2752719 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:07.235579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25280a1f-c107-4478-a0fa-9e72f2752719 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:07.265237 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-25280a1f-c107-4478-a0fa-9e72f2752719 None None] MFA Rules not processed for user `cee0bf1da3de453fb77c7018ada3c9d8`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:07:07.266552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-25280a1f-c107-4478-a0fa-9e72f2752719 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:07.334500 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1027/2069] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:07:07 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:07:07.346359 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced680c4-9867-4c53-a7d6-2f7f66f95e08 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:07.348486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ced680c4-9867-4c53-a7d6-2f7f66f95e08 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:07.374551 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:07:07.412097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced680c4-9867-4c53-a7d6-2f7f66f95e08 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:07.415627 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ccce1f4-cb11-473e-aa02-3a9ee56fdd46 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:07.416259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ccce1f4-cb11-473e-aa02-3a9ee56fdd46 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:07.416531 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ccce1f4-cb11-473e-aa02-3a9ee56fdd46 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:07.416807 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ccce1f4-cb11-473e-aa02-3a9ee56fdd46 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:07.424969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ccce1f4-cb11-473e-aa02-3a9ee56fdd46 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:07.426390 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ccce1f4-cb11-473e-aa02-3a9ee56fdd46 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:07.434861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1043/2070] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:07:07 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:07:07.466120 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25280a1f-c107-4478-a0fa-9e72f2752719 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:07.485894 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:07:07.529893 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25280a1f-c107-4478-a0fa-9e72f2752719 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:07.534470 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e69cc5-4cb2-4dcb-995a-a56e2386be3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:07.535099 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e69cc5-4cb2-4dcb-995a-a56e2386be3b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:07.535373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e69cc5-4cb2-4dcb-995a-a56e2386be3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:07.535638 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e69cc5-4cb2-4dcb-995a-a56e2386be3b admin admin] PATH_INFO: `/v3/users/cee0bf1da3de453fb77c7018ada3c9d8` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:07.536788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e69cc5-4cb2-4dcb-995a-a56e2386be3b admin admin] RBAC: Authorizing `identity:delete_user(user_id=cee0bf1da3de453fb77c7018ada3c9d8)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:07.538692 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e69cc5-4cb2-4dcb-995a-a56e2386be3b admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:07.587188 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-86e69cc5-4cb2-4dcb-995a-a56e2386be3b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cee0bf1da3de453fb77c7018ada3c9d8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:07.596067 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-86e69cc5-4cb2-4dcb-995a-a56e2386be3b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cee0bf1da3de453fb77c7018ada3c9d8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:07.596671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-86e69cc5-4cb2-4dcb-995a-a56e2386be3b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cee0bf1da3de453fb77c7018ada3c9d8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:07.603653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-86e69cc5-4cb2-4dcb-995a-a56e2386be3b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cee0bf1da3de453fb77c7018ada3c9d8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:07.615483 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-86e69cc5-4cb2-4dcb-995a-a56e2386be3b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cee0bf1da3de453fb77c7018ada3c9d8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:07.621253 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1028/2071] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:07:07 2022] DELETE /identity/v3/users/cee0bf1da3de453fb77c7018ada3c9d8 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:07:07.625879 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f679206-5eab-4152-85eb-75aeb863f577 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:07.626163 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f679206-5eab-4152-85eb-75aeb863f577 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:07.626425 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f679206-5eab-4152-85eb-75aeb863f577 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:07.645206 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-3f679206-5eab-4152-85eb-75aeb863f577 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:07:07.646394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3f679206-5eab-4152-85eb-75aeb863f577 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:07.706476 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1044/2072] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:07:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:07:07.716753 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e69cc5-4cb2-4dcb-995a-a56e2386be3b admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:07.718291 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-86e69cc5-4cb2-4dcb-995a-a56e2386be3b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:07.782315 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e69cc5-4cb2-4dcb-995a-a56e2386be3b admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:07.789176 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3555820d-a521-48a7-ac54-ceeac19dd740 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:07.789969 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3555820d-a521-48a7-ac54-ceeac19dd740 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:07.790348 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3555820d-a521-48a7-ac54-ceeac19dd740 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:07.790720 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3555820d-a521-48a7-ac54-ceeac19dd740 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:07.802207 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3555820d-a521-48a7-ac54-ceeac19dd740 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:07.803642 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3555820d-a521-48a7-ac54-ceeac19dd740 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:07.813551 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1029/2073] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:07:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:07:08.095203 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f679206-5eab-4152-85eb-75aeb863f577 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:08.096693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3f679206-5eab-4152-85eb-75aeb863f577 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:08.108502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f679206-5eab-4152-85eb-75aeb863f577 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:08.113245 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26035f97-26c7-4dc7-9efd-eb267fd75955 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:08.113886 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26035f97-26c7-4dc7-9efd-eb267fd75955 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:08.114159 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26035f97-26c7-4dc7-9efd-eb267fd75955 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:08.114426 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26035f97-26c7-4dc7-9efd-eb267fd75955 admin admin] PATH_INFO: `/v3/projects/55f89ce52f674f84973f4ea38be311a4` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:08.115574 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26035f97-26c7-4dc7-9efd-eb267fd75955 admin admin] RBAC: Authorizing `identity:delete_project(project_id=55f89ce52f674f84973f4ea38be311a4)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:08.117495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26035f97-26c7-4dc7-9efd-eb267fd75955 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:08.166289 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-26035f97-26c7-4dc7-9efd-eb267fd75955 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '55f89ce52f674f84973f4ea38be311a4', 'request_id': 'req-26035f97-26c7-4dc7-9efd-eb267fd75955'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:08.176352 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-26035f97-26c7-4dc7-9efd-eb267fd75955 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '55f89ce52f674f84973f4ea38be311a4', 'request_id': 'req-26035f97-26c7-4dc7-9efd-eb267fd75955'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:08.183602 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-26035f97-26c7-4dc7-9efd-eb267fd75955 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '55f89ce52f674f84973f4ea38be311a4', 'request_id': 'req-26035f97-26c7-4dc7-9efd-eb267fd75955'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:08.195907 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-26035f97-26c7-4dc7-9efd-eb267fd75955 admin admin] The token cache is being invalidate because project 55f89ce52f674f84973f4ea38be311a4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:07:08.196568 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-26035f97-26c7-4dc7-9efd-eb267fd75955 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:08.196908 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-26035f97-26c7-4dc7-9efd-eb267fd75955 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:08.198558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1045/2074] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:07:08 2022] DELETE /identity/v3/projects/55f89ce52f674f84973f4ea38be311a4 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:07:08.203847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e03c5b-a3f2-4ff9-91f0-78dd6d3fb26e None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:08.204281 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e03c5b-a3f2-4ff9-91f0-78dd6d3fb26e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:08.204402 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e03c5b-a3f2-4ff9-91f0-78dd6d3fb26e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:08.224458 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-87e03c5b-a3f2-4ff9-91f0-78dd6d3fb26e None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:07:08.225658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-87e03c5b-a3f2-4ff9-91f0-78dd6d3fb26e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:08.259877 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:07:08.288059 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1030/2075] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:07:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:07:08.295880 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6997c44e-6005-450b-8050-f1652bd1e33d None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:08.296167 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6997c44e-6005-450b-8050-f1652bd1e33d None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:08.296436 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6997c44e-6005-450b-8050-f1652bd1e33d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:08.317668 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-6997c44e-6005-450b-8050-f1652bd1e33d None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:07:08.318870 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-6997c44e-6005-450b-8050-f1652bd1e33d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:08.327458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1046/2076] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:07:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:07:08.332486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e03c5b-a3f2-4ff9-91f0-78dd6d3fb26e None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:08.346087 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e03c5b-a3f2-4ff9-91f0-78dd6d3fb26e None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:08.351186 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a1c23f-7859-40d9-8050-4a324b00b187 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:08.351828 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a1c23f-7859-40d9-8050-4a324b00b187 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:08.352108 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a1c23f-7859-40d9-8050-4a324b00b187 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:08.352408 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a1c23f-7859-40d9-8050-4a324b00b187 admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:08.353066 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a1c23f-7859-40d9-8050-4a324b00b187 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:07:08.353366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a1c23f-7859-40d9-8050-4a324b00b187 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:08.354960 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a1c23f-7859-40d9-8050-4a324b00b187 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:08.367878 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1031/2077] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:07:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 01:07:08.372692 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6997c44e-6005-450b-8050-f1652bd1e33d None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:08.382350 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6997c44e-6005-450b-8050-f1652bd1e33d None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:08.387640 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9350dbe3-2671-4cf4-a339-7d1ae0ee68ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:08.388379 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9350dbe3-2671-4cf4-a339-7d1ae0ee68ff admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:08.388667 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9350dbe3-2671-4cf4-a339-7d1ae0ee68ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:08.389042 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9350dbe3-2671-4cf4-a339-7d1ae0ee68ff admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:08.389670 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9350dbe3-2671-4cf4-a339-7d1ae0ee68ff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:08.391588 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9350dbe3-2671-4cf4-a339-7d1ae0ee68ff admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:08.424166 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1047/2078] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:07:08 2022] POST /identity/v3/projects => generated 377 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:07:08.425847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a1c23f-7859-40d9-8050-4a324b00b187 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:08.436238 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a1c23f-7859-40d9-8050-4a324b00b187 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:08.495129 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cddd560-3e03-4358-8bae-2a289313db7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:08.495807 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cddd560-3e03-4358-8bae-2a289313db7b admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:08.496053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cddd560-3e03-4358-8bae-2a289313db7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:08.496315 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cddd560-3e03-4358-8bae-2a289313db7b admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:08.496957 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cddd560-3e03-4358-8bae-2a289313db7b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:08.498990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cddd560-3e03-4358-8bae-2a289313db7b admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:08.523311 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1032/2079] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:07:08 2022] POST /identity/v3/users => generated 289 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:07:08.527418 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9350dbe3-2671-4cf4-a339-7d1ae0ee68ff admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:08.538362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9350dbe3-2671-4cf4-a339-7d1ae0ee68ff admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:08.542938 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf33b76-8cd2-4e91-a33a-728adf8166cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:08.543543 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf33b76-8cd2-4e91-a33a-728adf8166cf admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:08.543848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf33b76-8cd2-4e91-a33a-728adf8166cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:08.544117 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf33b76-8cd2-4e91-a33a-728adf8166cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:08.544827 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf33b76-8cd2-4e91-a33a-728adf8166cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:08.546577 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf33b76-8cd2-4e91-a33a-728adf8166cf admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:08.557877 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1048/2080] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:07:08 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:07:08.562069 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cddd560-3e03-4358-8bae-2a289313db7b admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:08.573061 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cddd560-3e03-4358-8bae-2a289313db7b admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:08.578152 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef3d09f-0618-4f46-aec5-41071905ecdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:08.578757 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef3d09f-0618-4f46-aec5-41071905ecdc admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:08.579029 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef3d09f-0618-4f46-aec5-41071905ecdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:08.579344 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef3d09f-0618-4f46-aec5-41071905ecdc admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:08.579968 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef3d09f-0618-4f46-aec5-41071905ecdc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:08.581549 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef3d09f-0618-4f46-aec5-41071905ecdc admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:08.592930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1033/2081] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:07:08 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:07:08.597428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf33b76-8cd2-4e91-a33a-728adf8166cf admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:08.608000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf33b76-8cd2-4e91-a33a-728adf8166cf admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:08.612993 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06e9d8a-5ed7-4b24-a1f9-4ee99044315e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:08.613650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06e9d8a-5ed7-4b24-a1f9-4ee99044315e admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:08.613927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06e9d8a-5ed7-4b24-a1f9-4ee99044315e admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:08.614251 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06e9d8a-5ed7-4b24-a1f9-4ee99044315e admin admin] PATH_INFO: `/v3/projects/238d71be4e094cb6b84f9a48a8a2c79c/users/96e0cd00e4ef4a2da960aa5d4f66972a/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:08.616658 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:07:08.634782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e06e9d8a-5ed7-4b24-a1f9-4ee99044315e admin admin] RBAC: Authorizing `identity:create_grant(project_id=238d71be4e094cb6b84f9a48a8a2c79c, user_id=96e0cd00e4ef4a2da960aa5d4f66972a, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:08.638966 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e06e9d8a-5ed7-4b24-a1f9-4ee99044315e admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:08.656874 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1049/2082] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:07:08 2022] PUT /identity/v3/projects/238d71be4e094cb6b84f9a48a8a2c79c/users/96e0cd00e4ef4a2da960aa5d4f66972a/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:07:08.662068 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18be60b-ee6c-45d3-bb5c-441405d517e0 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:08.662415 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18be60b-ee6c-45d3-bb5c-441405d517e0 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:08.662740 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18be60b-ee6c-45d3-bb5c-441405d517e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:08.682628 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-c18be60b-ee6c-45d3-bb5c-441405d517e0 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:07:08.683875 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-c18be60b-ee6c-45d3-bb5c-441405d517e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:08.771521 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1034/2083] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:07:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:07:08.784656 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06e9d8a-5ed7-4b24-a1f9-4ee99044315e admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:08.786439 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e06e9d8a-5ed7-4b24-a1f9-4ee99044315e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:08.861827 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06e9d8a-5ed7-4b24-a1f9-4ee99044315e admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:08.865392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263c08a0-3b3e-49b3-b395-f5bcb7a92d60 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:08.865994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263c08a0-3b3e-49b3-b395-f5bcb7a92d60 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:08.866264 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263c08a0-3b3e-49b3-b395-f5bcb7a92d60 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:08.866526 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263c08a0-3b3e-49b3-b395-f5bcb7a92d60 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:08.875249 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263c08a0-3b3e-49b3-b395-f5bcb7a92d60 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:08.876700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263c08a0-3b3e-49b3-b395-f5bcb7a92d60 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:08.885816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1050/2084] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:07:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:07:11.602861 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0519dd7-9d10-48f7-a83b-b838a663eda5 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:11.603386 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0519dd7-9d10-48f7-a83b-b838a663eda5 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:11.603386 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0519dd7-9d10-48f7-a83b-b838a663eda5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:11.638090 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-d0519dd7-9d10-48f7-a83b-b838a663eda5 None None] MFA Rules not processed for user `4716e7afe2784e2a9b872996d895c374`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:07:11.642141 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d0519dd7-9d10-48f7-a83b-b838a663eda5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:11.719808 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:07:11.743875 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1035/2085] 10.222.0.16 () {58 vars in 967 bytes} [Sun Oct 30 01:07:11 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:07:11.756975 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263c08a0-3b3e-49b3-b395-f5bcb7a92d60 service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:11.759863 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-263c08a0-3b3e-49b3-b395-f5bcb7a92d60 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:11.842393 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263c08a0-3b3e-49b3-b395-f5bcb7a92d60 service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:11.847520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042f5c95-59c3-4e0c-bc27-84a9e4377be5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '67f6eff8c0254cc6a210101dd7af988d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:11.848335 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042f5c95-59c3-4e0c-bc27-84a9e4377be5 service swift] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:11.848664 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042f5c95-59c3-4e0c-bc27-84a9e4377be5 service swift] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:11.848989 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042f5c95-59c3-4e0c-bc27-84a9e4377be5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:11.858210 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042f5c95-59c3-4e0c-bc27-84a9e4377be5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:11.859831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042f5c95-59c3-4e0c-bc27-84a9e4377be5 service swift] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:11.869154 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1051/2086] 10.222.0.16 () {60 vars in 1384 bytes} [Sun Oct 30 01:07:11 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 30 01:07:13.955123 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5b9fd2-3d5a-4598-a7c9-69d1a99cf16f None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:13.955631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5b9fd2-3d5a-4598-a7c9-69d1a99cf16f None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:13.955942 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5b9fd2-3d5a-4598-a7c9-69d1a99cf16f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:14.014504 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-cb5b9fd2-3d5a-4598-a7c9-69d1a99cf16f None None] MFA Rules not processed for user `96e0cd00e4ef4a2da960aa5d4f66972a`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:07:14.016385 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-cb5b9fd2-3d5a-4598-a7c9-69d1a99cf16f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:14.104018 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1036/2087] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:07:13 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:07:14.108956 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042f5c95-59c3-4e0c-bc27-84a9e4377be5 service swift] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:14.120165 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042f5c95-59c3-4e0c-bc27-84a9e4377be5 service swift] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:14.127500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1571c32-0acd-4535-94e3-10bff88db7d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:14.128781 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1571c32-0acd-4535-94e3-10bff88db7d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:14.129225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1571c32-0acd-4535-94e3-10bff88db7d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:14.129491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1571c32-0acd-4535-94e3-10bff88db7d3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:14.130280 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1571c32-0acd-4535-94e3-10bff88db7d3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:14.132304 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1571c32-0acd-4535-94e3-10bff88db7d3 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:14.184403 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1052/2088] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:07:14 2022] POST /identity/v3/projects => generated 379 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:07:14.185726 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5b9fd2-3d5a-4598-a7c9-69d1a99cf16f None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:14.204156 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5b9fd2-3d5a-4598-a7c9-69d1a99cf16f None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:14.285852 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdff4674-fe71-4077-889d-3db4017620b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:14.287190 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdff4674-fe71-4077-889d-3db4017620b4 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:14.287473 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdff4674-fe71-4077-889d-3db4017620b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:14.287776 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdff4674-fe71-4077-889d-3db4017620b4 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:14.288745 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdff4674-fe71-4077-889d-3db4017620b4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:14.292739 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdff4674-fe71-4077-889d-3db4017620b4 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:14.325078 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1037/2089] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:07:14 2022] POST /identity/v3/users => generated 296 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:07:14.328989 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1571c32-0acd-4535-94e3-10bff88db7d3 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:14.343206 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1571c32-0acd-4535-94e3-10bff88db7d3 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:14.360058 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175ccad8-a902-4540-82cf-db942c042703 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:14.360058 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175ccad8-a902-4540-82cf-db942c042703 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:14.360058 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175ccad8-a902-4540-82cf-db942c042703 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:14.360058 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175ccad8-a902-4540-82cf-db942c042703 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:14.360904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175ccad8-a902-4540-82cf-db942c042703 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:14.364628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175ccad8-a902-4540-82cf-db942c042703 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:14.380467 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1053/2090] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:07:14 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:07:14.384479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdff4674-fe71-4077-889d-3db4017620b4 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:14.396415 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdff4674-fe71-4077-889d-3db4017620b4 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:14.401425 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6770862b-d057-4afc-81c6-c4d0f8f508da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:14.402150 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6770862b-d057-4afc-81c6-c4d0f8f508da admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:14.402528 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6770862b-d057-4afc-81c6-c4d0f8f508da admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:14.402818 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6770862b-d057-4afc-81c6-c4d0f8f508da admin admin] PATH_INFO: `/v3/projects/39ed9d8c55cf4a549f2f49504650aa20/users/42de539be2df4b0082f01b3aa3746691/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:14.432061 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6770862b-d057-4afc-81c6-c4d0f8f508da admin admin] RBAC: Authorizing `identity:create_grant(project_id=39ed9d8c55cf4a549f2f49504650aa20, user_id=42de539be2df4b0082f01b3aa3746691, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:14.435009 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6770862b-d057-4afc-81c6-c4d0f8f508da admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:14.452846 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1038/2091] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:07:14 2022] PUT /identity/v3/projects/39ed9d8c55cf4a549f2f49504650aa20/users/42de539be2df4b0082f01b3aa3746691/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:07:14.457075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175ccad8-a902-4540-82cf-db942c042703 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:14.468695 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175ccad8-a902-4540-82cf-db942c042703 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:14.513843 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:07:14.535700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1faffd-724b-4f1b-bdec-75d5736c81ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:14.536347 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1faffd-724b-4f1b-bdec-75d5736c81ee admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:14.536617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1faffd-724b-4f1b-bdec-75d5736c81ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:14.536879 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1faffd-724b-4f1b-bdec-75d5736c81ee admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:14.537517 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1faffd-724b-4f1b-bdec-75d5736c81ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:14.539397 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1faffd-724b-4f1b-bdec-75d5736c81ee admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:14.552440 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1054/2092] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:07:14 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:07:14.556919 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6770862b-d057-4afc-81c6-c4d0f8f508da admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:14.573013 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6770862b-d057-4afc-81c6-c4d0f8f508da admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:14.583768 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f686eda-0755-4c06-a9a4-c3022bb3e8fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:14.584602 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f686eda-0755-4c06-a9a4-c3022bb3e8fb admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:14.584602 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f686eda-0755-4c06-a9a4-c3022bb3e8fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:14.586006 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f686eda-0755-4c06-a9a4-c3022bb3e8fb admin admin] PATH_INFO: `/v3/projects/39ed9d8c55cf4a549f2f49504650aa20/users/42de539be2df4b0082f01b3aa3746691/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:14.586909 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f686eda-0755-4c06-a9a4-c3022bb3e8fb admin admin] RBAC: Authorizing `identity:create_grant(project_id=39ed9d8c55cf4a549f2f49504650aa20, user_id=42de539be2df4b0082f01b3aa3746691, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:14.589928 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f686eda-0755-4c06-a9a4-c3022bb3e8fb admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:14.607888 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1039/2093] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:07:14 2022] PUT /identity/v3/projects/39ed9d8c55cf4a549f2f49504650aa20/users/42de539be2df4b0082f01b3aa3746691/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:07:14.630393 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1faffd-724b-4f1b-bdec-75d5736c81ee admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:14.632444 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ba1faffd-724b-4f1b-bdec-75d5736c81ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:14.747361 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1faffd-724b-4f1b-bdec-75d5736c81ee admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:14.822419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993b7b0f-53ef-4135-ad95-355900a62d05 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:14.823060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993b7b0f-53ef-4135-ad95-355900a62d05 service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:14.823334 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993b7b0f-53ef-4135-ad95-355900a62d05 service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:14.823598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993b7b0f-53ef-4135-ad95-355900a62d05 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:14.825945 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-993b7b0f-53ef-4135-ad95-355900a62d05 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:14.841774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993b7b0f-53ef-4135-ad95-355900a62d05 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:14.844326 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993b7b0f-53ef-4135-ad95-355900a62d05 service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:14.871823 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1055/2094] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:07:14 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 234 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:07:16.627501 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13d4e58-c8cc-42ff-b844-96d188e01805 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:16.628050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13d4e58-c8cc-42ff-b844-96d188e01805 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:16.628050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13d4e58-c8cc-42ff-b844-96d188e01805 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:16.652099 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-f13d4e58-c8cc-42ff-b844-96d188e01805 None None] MFA Rules not processed for user `4716e7afe2784e2a9b872996d895c374`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:07:16.653705 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-f13d4e58-c8cc-42ff-b844-96d188e01805 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:16.689147 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:07:16.732124 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1040/2095] 10.222.0.16 () {58 vars in 967 bytes} [Sun Oct 30 01:07:16 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:07:16.739942 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993b7b0f-53ef-4135-ad95-355900a62d05 service nova] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:16.806604 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:07:16.811386 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993b7b0f-53ef-4135-ad95-355900a62d05 service nova] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:16.817509 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bd507e-7472-4a6a-bc57-f801d0a8b6de service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '67f6eff8c0254cc6a210101dd7af988d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:16.818189 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bd507e-7472-4a6a-bc57-f801d0a8b6de service swift] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:16.818456 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bd507e-7472-4a6a-bc57-f801d0a8b6de service swift] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:16.818719 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bd507e-7472-4a6a-bc57-f801d0a8b6de service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:16.828094 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bd507e-7472-4a6a-bc57-f801d0a8b6de service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:16.829078 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bd507e-7472-4a6a-bc57-f801d0a8b6de service swift] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:16.837626 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1056/2096] 10.222.0.16 () {60 vars in 1384 bytes} [Sun Oct 30 01:07:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 30 01:07:19.647400 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07446a4-4747-4a36-b4a0-62fe199743df None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:19.647851 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07446a4-4747-4a36-b4a0-62fe199743df None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:19.648603 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07446a4-4747-4a36-b4a0-62fe199743df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:19.709302 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-a07446a4-4747-4a36-b4a0-62fe199743df None None] MFA Rules not processed for user `42de539be2df4b0082f01b3aa3746691`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:07:19.710682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a07446a4-4747-4a36-b4a0-62fe199743df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:19.848532 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1041/2097] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:07:19 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:07:19.862818 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bd507e-7472-4a6a-bc57-f801d0a8b6de service swift] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:19.880287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bd507e-7472-4a6a-bc57-f801d0a8b6de service swift] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:19.895242 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bd9f16-0c69-4c07-acbe-8aeb629dd94b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:19.896274 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bd9f16-0c69-4c07-acbe-8aeb629dd94b service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:19.896744 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bd9f16-0c69-4c07-acbe-8aeb629dd94b service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:19.897419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bd9f16-0c69-4c07-acbe-8aeb629dd94b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:20.014789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bd9f16-0c69-4c07-acbe-8aeb629dd94b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:20.016876 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bd9f16-0c69-4c07-acbe-8aeb629dd94b service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:20.029601 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1057/2098] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:07:19 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:07:25.748584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07446a4-4747-4a36-b4a0-62fe199743df None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:25.750107 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a07446a4-4747-4a36-b4a0-62fe199743df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:25.841500 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07446a4-4747-4a36-b4a0-62fe199743df None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:25.847629 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c993d1-2124-4ecc-88e2-763fd63384b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:25.848330 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c993d1-2124-4ecc-88e2-763fd63384b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:25.848603 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c993d1-2124-4ecc-88e2-763fd63384b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:25.848893 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c993d1-2124-4ecc-88e2-763fd63384b2 admin admin] PATH_INFO: `/v3/users/05eed70effab4e38ae469ab44d1114db` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:25.850015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c993d1-2124-4ecc-88e2-763fd63384b2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=05eed70effab4e38ae469ab44d1114db)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:25.851998 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c993d1-2124-4ecc-88e2-763fd63384b2 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:25.910870 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b2c993d1-2124-4ecc-88e2-763fd63384b2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '05eed70effab4e38ae469ab44d1114db'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:25.925349 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b2c993d1-2124-4ecc-88e2-763fd63384b2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '05eed70effab4e38ae469ab44d1114db'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:25.926027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b2c993d1-2124-4ecc-88e2-763fd63384b2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '05eed70effab4e38ae469ab44d1114db'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:25.934656 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b2c993d1-2124-4ecc-88e2-763fd63384b2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '05eed70effab4e38ae469ab44d1114db'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:25.939877 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:07:25.971896 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b2c993d1-2124-4ecc-88e2-763fd63384b2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '05eed70effab4e38ae469ab44d1114db'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:25.981348 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1042/2099] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:07:25 2022] DELETE /identity/v3/users/05eed70effab4e38ae469ab44d1114db => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:07:26.176488 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bd9f16-0c69-4c07-acbe-8aeb629dd94b service nova] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:26.188040 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bd9f16-0c69-4c07-acbe-8aeb629dd94b service nova] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:26.258774 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd126e86-a3af-4582-a989-fcf01bbfc697 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:26.259536 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd126e86-a3af-4582-a989-fcf01bbfc697 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:26.259847 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd126e86-a3af-4582-a989-fcf01bbfc697 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:26.260114 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd126e86-a3af-4582-a989-fcf01bbfc697 admin admin] PATH_INFO: `/v3/projects/1dcbf2e9fb2e4a4698633fcbb07c30cd` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:26.261468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd126e86-a3af-4582-a989-fcf01bbfc697 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1dcbf2e9fb2e4a4698633fcbb07c30cd)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:26.263445 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd126e86-a3af-4582-a989-fcf01bbfc697 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:26.316092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-bd126e86-a3af-4582-a989-fcf01bbfc697 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1dcbf2e9fb2e4a4698633fcbb07c30cd', 'request_id': 'req-bd126e86-a3af-4582-a989-fcf01bbfc697'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:26.328187 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-bd126e86-a3af-4582-a989-fcf01bbfc697 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1dcbf2e9fb2e4a4698633fcbb07c30cd', 'request_id': 'req-bd126e86-a3af-4582-a989-fcf01bbfc697'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:26.335395 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-bd126e86-a3af-4582-a989-fcf01bbfc697 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1dcbf2e9fb2e4a4698633fcbb07c30cd', 'request_id': 'req-bd126e86-a3af-4582-a989-fcf01bbfc697'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:26.354065 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-bd126e86-a3af-4582-a989-fcf01bbfc697 admin admin] The token cache is being invalidate because project 1dcbf2e9fb2e4a4698633fcbb07c30cd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:07:26.354555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-bd126e86-a3af-4582-a989-fcf01bbfc697 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:26.358456 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-bd126e86-a3af-4582-a989-fcf01bbfc697 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:26.360762 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1058/2100] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:07:26 2022] DELETE /identity/v3/projects/1dcbf2e9fb2e4a4698633fcbb07c30cd => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:07:26.364482 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c993d1-2124-4ecc-88e2-763fd63384b2 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:26.365939 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b2c993d1-2124-4ecc-88e2-763fd63384b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:26.383523 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c993d1-2124-4ecc-88e2-763fd63384b2 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:26.466162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcea239-2613-4918-8426-008d9db6a001 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:26.466820 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcea239-2613-4918-8426-008d9db6a001 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:26.467256 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcea239-2613-4918-8426-008d9db6a001 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:26.467481 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcea239-2613-4918-8426-008d9db6a001 admin admin] PATH_INFO: `/v3/users/7e83347263024b759e9a65e971336335` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:26.468698 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfcea239-2613-4918-8426-008d9db6a001 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7e83347263024b759e9a65e971336335)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:26.470738 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfcea239-2613-4918-8426-008d9db6a001 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:26.537444 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-cfcea239-2613-4918-8426-008d9db6a001 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7e83347263024b759e9a65e971336335'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:26.548268 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-cfcea239-2613-4918-8426-008d9db6a001 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7e83347263024b759e9a65e971336335'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:26.549146 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-cfcea239-2613-4918-8426-008d9db6a001 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7e83347263024b759e9a65e971336335'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:26.562108 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-cfcea239-2613-4918-8426-008d9db6a001 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7e83347263024b759e9a65e971336335'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:26.576671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-cfcea239-2613-4918-8426-008d9db6a001 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7e83347263024b759e9a65e971336335'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:26.584188 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1043/2101] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:07:26 2022] DELETE /identity/v3/users/7e83347263024b759e9a65e971336335 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:07:27.067153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd126e86-a3af-4582-a989-fcf01bbfc697 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:27.069115 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-bd126e86-a3af-4582-a989-fcf01bbfc697 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:27.082472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd126e86-a3af-4582-a989-fcf01bbfc697 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:27.102470 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:07:27.146580 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b667f60-3064-4df7-9216-172a89a1e10c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:27.147226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b667f60-3064-4df7-9216-172a89a1e10c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:27.147500 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b667f60-3064-4df7-9216-172a89a1e10c admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:27.147804 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b667f60-3064-4df7-9216-172a89a1e10c admin admin] PATH_INFO: `/v3/projects/a2e086e391bc4489bba3032b906ca722` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:27.148926 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b667f60-3064-4df7-9216-172a89a1e10c admin admin] RBAC: Authorizing `identity:delete_project(project_id=a2e086e391bc4489bba3032b906ca722)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:27.150867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b667f60-3064-4df7-9216-172a89a1e10c admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:27.203944 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-2b667f60-3064-4df7-9216-172a89a1e10c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a2e086e391bc4489bba3032b906ca722', 'request_id': 'req-2b667f60-3064-4df7-9216-172a89a1e10c'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:27.212904 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-2b667f60-3064-4df7-9216-172a89a1e10c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a2e086e391bc4489bba3032b906ca722', 'request_id': 'req-2b667f60-3064-4df7-9216-172a89a1e10c'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:27.219892 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-2b667f60-3064-4df7-9216-172a89a1e10c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a2e086e391bc4489bba3032b906ca722', 'request_id': 'req-2b667f60-3064-4df7-9216-172a89a1e10c'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:27.232253 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-2b667f60-3064-4df7-9216-172a89a1e10c admin admin] The token cache is being invalidate because project a2e086e391bc4489bba3032b906ca722 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:07:27.232542 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-2b667f60-3064-4df7-9216-172a89a1e10c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:27.233322 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-2b667f60-3064-4df7-9216-172a89a1e10c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:27.234977 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1059/2102] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:07:27 2022] DELETE /identity/v3/projects/a2e086e391bc4489bba3032b906ca722 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:07:36.579351 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=59,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:07:37.228819 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=42,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:07:43.044199 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcea239-2613-4918-8426-008d9db6a001 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:43.045720 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-cfcea239-2613-4918-8426-008d9db6a001 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:43.107882 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcea239-2613-4918-8426-008d9db6a001 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:43.166656 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19ce7a2-8e9d-481c-a3cf-6f7ef0cea8ff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:43.167181 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19ce7a2-8e9d-481c-a3cf-6f7ef0cea8ff service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:43.167453 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19ce7a2-8e9d-481c-a3cf-6f7ef0cea8ff service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:43.167754 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19ce7a2-8e9d-481c-a3cf-6f7ef0cea8ff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:43.169657 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b19ce7a2-8e9d-481c-a3cf-6f7ef0cea8ff service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:43.240923 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19ce7a2-8e9d-481c-a3cf-6f7ef0cea8ff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:43.242371 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19ce7a2-8e9d-481c-a3cf-6f7ef0cea8ff service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:43.252090 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1044/2103] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:07:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:07:46.432110 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b667f60-3064-4df7-9216-172a89a1e10c admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:46.434016 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-2b667f60-3064-4df7-9216-172a89a1e10c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:46.514220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19ce7a2-8e9d-481c-a3cf-6f7ef0cea8ff service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:46.514220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b19ce7a2-8e9d-481c-a3cf-6f7ef0cea8ff service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:46.515242 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b667f60-3064-4df7-9216-172a89a1e10c admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:46.520925 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8c10b3-5c1a-4e6b-bbed-50bc33bf54f4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '63cc035057064805a95e9f12714ebade', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:46.522714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8c10b3-5c1a-4e6b-bbed-50bc33bf54f4 service placement] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:46.522714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8c10b3-5c1a-4e6b-bbed-50bc33bf54f4 service placement] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:46.522714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8c10b3-5c1a-4e6b-bbed-50bc33bf54f4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:46.525112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-2e8c10b3-5c1a-4e6b-bbed-50bc33bf54f4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:46.527104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19ce7a2-8e9d-481c-a3cf-6f7ef0cea8ff service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:46.532303 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57883899-4273-4aa3-af0d-7fe36b52cdf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:46.532946 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57883899-4273-4aa3-af0d-7fe36b52cdf6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:46.533248 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57883899-4273-4aa3-af0d-7fe36b52cdf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:46.533479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57883899-4273-4aa3-af0d-7fe36b52cdf6 admin admin] PATH_INFO: `/v3/users/1c0e5bd11ac14259ad376ce54bcb7025` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:46.534815 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57883899-4273-4aa3-af0d-7fe36b52cdf6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1c0e5bd11ac14259ad376ce54bcb7025)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:46.537004 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57883899-4273-4aa3-af0d-7fe36b52cdf6 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:46.538081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8c10b3-5c1a-4e6b-bbed-50bc33bf54f4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:46.539766 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8c10b3-5c1a-4e6b-bbed-50bc33bf54f4 service placement] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:46.550569 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1060/2104] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:07:46 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:07:46.591355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-57883899-4273-4aa3-af0d-7fe36b52cdf6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1c0e5bd11ac14259ad376ce54bcb7025'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:46.600036 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-57883899-4273-4aa3-af0d-7fe36b52cdf6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1c0e5bd11ac14259ad376ce54bcb7025'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:46.600624 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-57883899-4273-4aa3-af0d-7fe36b52cdf6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1c0e5bd11ac14259ad376ce54bcb7025'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:46.606041 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:07:46.608165 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-57883899-4273-4aa3-af0d-7fe36b52cdf6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1c0e5bd11ac14259ad376ce54bcb7025'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:46.620983 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-57883899-4273-4aa3-af0d-7fe36b52cdf6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1c0e5bd11ac14259ad376ce54bcb7025'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:46.626902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1045/2105] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:07:46 2022] DELETE /identity/v3/users/1c0e5bd11ac14259ad376ce54bcb7025 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:07:46.799709 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8c10b3-5c1a-4e6b-bbed-50bc33bf54f4 service placement] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:46.814647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8c10b3-5c1a-4e6b-bbed-50bc33bf54f4 service placement] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:46.876021 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8cc918b-44ac-41ba-8cdf-122773585978 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:46.876724 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cc918b-44ac-41ba-8cdf-122773585978 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:46.877004 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cc918b-44ac-41ba-8cdf-122773585978 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:46.877270 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cc918b-44ac-41ba-8cdf-122773585978 admin admin] PATH_INFO: `/v3/projects/8bf407954ed34ebfaf56db206df4fc84` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:46.878471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8cc918b-44ac-41ba-8cdf-122773585978 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8bf407954ed34ebfaf56db206df4fc84)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:46.880457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8cc918b-44ac-41ba-8cdf-122773585978 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:46.929198 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c8cc918b-44ac-41ba-8cdf-122773585978 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8bf407954ed34ebfaf56db206df4fc84', 'request_id': 'req-c8cc918b-44ac-41ba-8cdf-122773585978'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:46.939008 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c8cc918b-44ac-41ba-8cdf-122773585978 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8bf407954ed34ebfaf56db206df4fc84', 'request_id': 'req-c8cc918b-44ac-41ba-8cdf-122773585978'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:46.944626 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c8cc918b-44ac-41ba-8cdf-122773585978 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8bf407954ed34ebfaf56db206df4fc84', 'request_id': 'req-c8cc918b-44ac-41ba-8cdf-122773585978'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:46.956221 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c8cc918b-44ac-41ba-8cdf-122773585978 admin admin] The token cache is being invalidate because project 8bf407954ed34ebfaf56db206df4fc84 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:07:46.956501 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c8cc918b-44ac-41ba-8cdf-122773585978 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:46.957141 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c8cc918b-44ac-41ba-8cdf-122773585978 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:46.958726 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1061/2106] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:07:46 2022] DELETE /identity/v3/projects/8bf407954ed34ebfaf56db206df4fc84 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:07:46.965317 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe30533-add0-435f-8776-5750663e676d None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:46.965597 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe30533-add0-435f-8776-5750663e676d None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:46.965866 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe30533-add0-435f-8776-5750663e676d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:46.986634 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-7fe30533-add0-435f-8776-5750663e676d None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:07:46.987875 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-7fe30533-add0-435f-8776-5750663e676d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:47.047175 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1046/2107] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:07:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:07:47.054984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6036403b-b4c3-4b96-b3b7-c872b7ef02c3 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:47.055259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6036403b-b4c3-4b96-b3b7-c872b7ef02c3 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:47.055525 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6036403b-b4c3-4b96-b3b7-c872b7ef02c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:47.075297 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-6036403b-b4c3-4b96-b3b7-c872b7ef02c3 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:07:47.076505 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-6036403b-b4c3-4b96-b3b7-c872b7ef02c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:47.084709 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1062/2108] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:07:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:07:47.089339 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe30533-add0-435f-8776-5750663e676d None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:47.098897 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe30533-add0-435f-8776-5750663e676d None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:47.103411 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ac1ef1-61c2-450f-b309-694471b5bff3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:47.104057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ac1ef1-61c2-450f-b309-694471b5bff3 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:47.104330 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ac1ef1-61c2-450f-b309-694471b5bff3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:47.104593 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ac1ef1-61c2-450f-b309-694471b5bff3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:47.105257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5ac1ef1-61c2-450f-b309-694471b5bff3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:07:47.105551 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5ac1ef1-61c2-450f-b309-694471b5bff3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:47.107152 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5ac1ef1-61c2-450f-b309-694471b5bff3 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:47.118589 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1047/2109] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:07:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 01:07:47.122851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6036403b-b4c3-4b96-b3b7-c872b7ef02c3 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:47.132682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6036403b-b4c3-4b96-b3b7-c872b7ef02c3 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:47.137518 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91f20ce-cc32-41d8-9900-f66b7e1e2b79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:47.138153 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91f20ce-cc32-41d8-9900-f66b7e1e2b79 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:47.138422 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91f20ce-cc32-41d8-9900-f66b7e1e2b79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:47.138659 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91f20ce-cc32-41d8-9900-f66b7e1e2b79 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:47.139372 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91f20ce-cc32-41d8-9900-f66b7e1e2b79 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:47.141419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91f20ce-cc32-41d8-9900-f66b7e1e2b79 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:47.152750 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:07:47.174013 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1063/2110] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:07:47 2022] POST /identity/v3/projects => generated 385 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:07:47.178515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ac1ef1-61c2-450f-b309-694471b5bff3 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:47.189103 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ac1ef1-61c2-450f-b309-694471b5bff3 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:47.246361 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef0d4d4-cc29-42d6-8be4-bb7a632cb4e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:47.246985 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef0d4d4-cc29-42d6-8be4-bb7a632cb4e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:47.247232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef0d4d4-cc29-42d6-8be4-bb7a632cb4e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:47.247496 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef0d4d4-cc29-42d6-8be4-bb7a632cb4e2 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:47.248156 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef0d4d4-cc29-42d6-8be4-bb7a632cb4e2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:47.250181 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef0d4d4-cc29-42d6-8be4-bb7a632cb4e2 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:47.275197 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1048/2111] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:07:47 2022] POST /identity/v3/users => generated 293 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:07:47.279969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91f20ce-cc32-41d8-9900-f66b7e1e2b79 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:47.290658 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91f20ce-cc32-41d8-9900-f66b7e1e2b79 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:47.296477 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c865eb30-4661-4d79-821f-c84c8d2e6555 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:47.297073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c865eb30-4661-4d79-821f-c84c8d2e6555 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:47.297430 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c865eb30-4661-4d79-821f-c84c8d2e6555 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:47.297585 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c865eb30-4661-4d79-821f-c84c8d2e6555 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:47.298211 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c865eb30-4661-4d79-821f-c84c8d2e6555 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:47.300109 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c865eb30-4661-4d79-821f-c84c8d2e6555 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:47.310382 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1064/2112] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:07:47 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:07:47.314328 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef0d4d4-cc29-42d6-8be4-bb7a632cb4e2 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:47.324876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef0d4d4-cc29-42d6-8be4-bb7a632cb4e2 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:47.331119 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e8c240-a561-4d4b-ac78-b2c1dcd9dcff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:47.331743 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e8c240-a561-4d4b-ac78-b2c1dcd9dcff admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:47.331992 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e8c240-a561-4d4b-ac78-b2c1dcd9dcff admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:47.332254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e8c240-a561-4d4b-ac78-b2c1dcd9dcff admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:47.332891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e8c240-a561-4d4b-ac78-b2c1dcd9dcff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:47.334480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e8c240-a561-4d4b-ac78-b2c1dcd9dcff admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:47.346348 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1049/2113] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:07:47 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:07:47.350840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c865eb30-4661-4d79-821f-c84c8d2e6555 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:47.361205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c865eb30-4661-4d79-821f-c84c8d2e6555 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:47.367184 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e010f38e-6c75-4ad4-a7cf-868be299e6cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:47.367927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e010f38e-6c75-4ad4-a7cf-868be299e6cc admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:47.368199 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e010f38e-6c75-4ad4-a7cf-868be299e6cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:47.368474 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e010f38e-6c75-4ad4-a7cf-868be299e6cc admin admin] PATH_INFO: `/v3/projects/ed21a370b0bb43698b8aa11334ceaad9/users/dfff736ef7ea45a6a1dd3715f23c6041/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:47.384214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e010f38e-6c75-4ad4-a7cf-868be299e6cc admin admin] RBAC: Authorizing `identity:create_grant(project_id=ed21a370b0bb43698b8aa11334ceaad9, user_id=dfff736ef7ea45a6a1dd3715f23c6041, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:47.386914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e010f38e-6c75-4ad4-a7cf-868be299e6cc admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:47.401590 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1065/2114] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:07:47 2022] PUT /identity/v3/projects/ed21a370b0bb43698b8aa11334ceaad9/users/dfff736ef7ea45a6a1dd3715f23c6041/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:07:47.412394 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf19c81-d447-4412-972c-be964bf93c5c None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:47.412672 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf19c81-d447-4412-972c-be964bf93c5c None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:47.412941 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf19c81-d447-4412-972c-be964bf93c5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:47.441274 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-9bf19c81-d447-4412-972c-be964bf93c5c None None] MFA Rules not processed for user `dfff736ef7ea45a6a1dd3715f23c6041`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:07:47.442464 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-9bf19c81-d447-4412-972c-be964bf93c5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:47.475545 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:07:47.509807 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1050/2115] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:07:47 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:07:47.514638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e010f38e-6c75-4ad4-a7cf-868be299e6cc admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:47.579773 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e010f38e-6c75-4ad4-a7cf-868be299e6cc admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:47.584540 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e27e62-daec-44af-bd38-bd9e133a7971 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:47.585143 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e27e62-daec-44af-bd38-bd9e133a7971 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:47.585415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e27e62-daec-44af-bd38-bd9e133a7971 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:47.585683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e27e62-daec-44af-bd38-bd9e133a7971 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:47.586297 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e27e62-daec-44af-bd38-bd9e133a7971 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:47.588577 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e27e62-daec-44af-bd38-bd9e133a7971 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:47.617480 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1066/2116] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:07:47 2022] POST /identity/v3/projects => generated 385 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:07:47.623116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf19c81-d447-4412-972c-be964bf93c5c None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:47.639352 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf19c81-d447-4412-972c-be964bf93c5c None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:47.705077 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7cf1ddf-ede8-482d-b3a0-de52c6d35fed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:47.705762 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cf1ddf-ede8-482d-b3a0-de52c6d35fed admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:47.706044 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cf1ddf-ede8-482d-b3a0-de52c6d35fed admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:47.706311 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cf1ddf-ede8-482d-b3a0-de52c6d35fed admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:47.706962 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7cf1ddf-ede8-482d-b3a0-de52c6d35fed admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:47.709023 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7cf1ddf-ede8-482d-b3a0-de52c6d35fed admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:47.732710 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1051/2117] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:07:47 2022] POST /identity/v3/users => generated 299 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:07:47.736893 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e27e62-daec-44af-bd38-bd9e133a7971 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:47.747337 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e27e62-daec-44af-bd38-bd9e133a7971 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:47.752215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59eaca7b-d04e-4c8c-8375-e8fd09215cc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:47.752806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59eaca7b-d04e-4c8c-8375-e8fd09215cc3 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:47.753077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59eaca7b-d04e-4c8c-8375-e8fd09215cc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:47.753347 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59eaca7b-d04e-4c8c-8375-e8fd09215cc3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:47.753974 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59eaca7b-d04e-4c8c-8375-e8fd09215cc3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:47.755543 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59eaca7b-d04e-4c8c-8375-e8fd09215cc3 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:47.766628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1067/2118] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:07:47 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:07:47.771002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7cf1ddf-ede8-482d-b3a0-de52c6d35fed admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:47.782087 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7cf1ddf-ede8-482d-b3a0-de52c6d35fed admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:47.787422 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3d5e68-9521-495f-a555-5ebb00083e05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:47.788107 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3d5e68-9521-495f-a555-5ebb00083e05 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:47.788376 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3d5e68-9521-495f-a555-5ebb00083e05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:47.788638 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3d5e68-9521-495f-a555-5ebb00083e05 admin admin] PATH_INFO: `/v3/projects/f2de7c6c60584571a3a57f1678a24e64/users/81ed6d85c63747949353c1c8c75ffec4/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:47.805882 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3d5e68-9521-495f-a555-5ebb00083e05 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f2de7c6c60584571a3a57f1678a24e64, user_id=81ed6d85c63747949353c1c8c75ffec4, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:47.808712 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3d5e68-9521-495f-a555-5ebb00083e05 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:47.823927 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1052/2119] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:07:47 2022] PUT /identity/v3/projects/f2de7c6c60584571a3a57f1678a24e64/users/81ed6d85c63747949353c1c8c75ffec4/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:07:47.829516 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59eaca7b-d04e-4c8c-8375-e8fd09215cc3 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:47.840312 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59eaca7b-d04e-4c8c-8375-e8fd09215cc3 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:47.911410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad5079a-c6b0-4718-a6aa-b37be92d59c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:47.912371 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad5079a-c6b0-4718-a6aa-b37be92d59c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:47.912810 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad5079a-c6b0-4718-a6aa-b37be92d59c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:47.913254 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad5079a-c6b0-4718-a6aa-b37be92d59c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:47.914236 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad5079a-c6b0-4718-a6aa-b37be92d59c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:47.916846 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad5079a-c6b0-4718-a6aa-b37be92d59c1 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:47.931727 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1068/2120] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:07:47 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:07:47.936275 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3d5e68-9521-495f-a555-5ebb00083e05 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:47.947747 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3d5e68-9521-495f-a555-5ebb00083e05 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:47.953583 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c568f8d6-af10-4225-8a75-eb1e98950a37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:47.954276 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c568f8d6-af10-4225-8a75-eb1e98950a37 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:47.954545 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c568f8d6-af10-4225-8a75-eb1e98950a37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:47.954822 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c568f8d6-af10-4225-8a75-eb1e98950a37 admin admin] PATH_INFO: `/v3/projects/f2de7c6c60584571a3a57f1678a24e64/users/81ed6d85c63747949353c1c8c75ffec4/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:47.957040 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c568f8d6-af10-4225-8a75-eb1e98950a37 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f2de7c6c60584571a3a57f1678a24e64, user_id=81ed6d85c63747949353c1c8c75ffec4, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:47.959972 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c568f8d6-af10-4225-8a75-eb1e98950a37 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:47.972395 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1053/2121] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:07:47 2022] PUT /identity/v3/projects/f2de7c6c60584571a3a57f1678a24e64/users/81ed6d85c63747949353c1c8c75ffec4/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:07:47.985715 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7f12f5-2324-4bb6-8355-f70b814712cc None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:47.986094 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7f12f5-2324-4bb6-8355-f70b814712cc None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:47.986543 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7f12f5-2324-4bb6-8355-f70b814712cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:48.027008 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-4a7f12f5-2324-4bb6-8355-f70b814712cc None None] MFA Rules not processed for user `81ed6d85c63747949353c1c8c75ffec4`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:07:48.028447 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-4a7f12f5-2324-4bb6-8355-f70b814712cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:48.046093 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:07:48.118331 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1069/2122] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:07:47 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:07:48.132856 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c568f8d6-af10-4225-8a75-eb1e98950a37 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:48.136346 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-c568f8d6-af10-4225-8a75-eb1e98950a37 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:48.204901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c568f8d6-af10-4225-8a75-eb1e98950a37 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:48.208685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657c6cd5-dd70-4686-b462-2baeb41b1115 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:48.209293 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657c6cd5-dd70-4686-b462-2baeb41b1115 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:48.209564 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657c6cd5-dd70-4686-b462-2baeb41b1115 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:48.209826 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657c6cd5-dd70-4686-b462-2baeb41b1115 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:48.258260 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:07:48.279465 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657c6cd5-dd70-4686-b462-2baeb41b1115 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:48.281154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657c6cd5-dd70-4686-b462-2baeb41b1115 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:48.291541 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1054/2123] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:07:48 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:07:50.416195 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2520de-c5d9-4d68-9709-619b17527659 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:50.416727 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2520de-c5d9-4d68-9709-619b17527659 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:50.416727 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2520de-c5d9-4d68-9709-619b17527659 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:50.438062 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-8b2520de-c5d9-4d68-9709-619b17527659 None None] MFA Rules not processed for user `96e0cd00e4ef4a2da960aa5d4f66972a`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:07:50.439252 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-8b2520de-c5d9-4d68-9709-619b17527659 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:50.502755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1070/2124] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:07:50 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:07:50.828724 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657c6cd5-dd70-4686-b462-2baeb41b1115 service cinder] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:50.830150 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-657c6cd5-dd70-4686-b462-2baeb41b1115 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:50.888646 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657c6cd5-dd70-4686-b462-2baeb41b1115 service cinder] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:50.892409 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29d542a-54b7-4ae1-9ac8-c1443e591ba1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:50.893028 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29d542a-54b7-4ae1-9ac8-c1443e591ba1 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:50.893269 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29d542a-54b7-4ae1-9ac8-c1443e591ba1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:50.893553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29d542a-54b7-4ae1-9ac8-c1443e591ba1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:50.895288 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-f29d542a-54b7-4ae1-9ac8-c1443e591ba1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:50.905052 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29d542a-54b7-4ae1-9ac8-c1443e591ba1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:50.906494 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29d542a-54b7-4ae1-9ac8-c1443e591ba1 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:50.914465 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1055/2125] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:07:50 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:07:53.701435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2520de-c5d9-4d68-9709-619b17527659 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:53.703240 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-8b2520de-c5d9-4d68-9709-619b17527659 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:53.716378 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2520de-c5d9-4d68-9709-619b17527659 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:53.720102 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31958915-8891-4b07-b9e2-c86a9de68ceb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:53.720950 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31958915-8891-4b07-b9e2-c86a9de68ceb service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:53.721373 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31958915-8891-4b07-b9e2-c86a9de68ceb service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:53.721651 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31958915-8891-4b07-b9e2-c86a9de68ceb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:53.731190 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31958915-8891-4b07-b9e2-c86a9de68ceb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:53.732652 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31958915-8891-4b07-b9e2-c86a9de68ceb service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:53.743135 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1071/2126] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:07:53 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:07:58.701322 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29d542a-54b7-4ae1-9ac8-c1443e591ba1 service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:58.767379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29d542a-54b7-4ae1-9ac8-c1443e591ba1 service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:58.772461 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1bde20-190d-4b23-8f22-84e443fcc7ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:58.773096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1bde20-190d-4b23-8f22-84e443fcc7ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:58.773366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1bde20-190d-4b23-8f22-84e443fcc7ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:58.773630 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1bde20-190d-4b23-8f22-84e443fcc7ea admin admin] PATH_INFO: `/v3/users/dfff736ef7ea45a6a1dd3715f23c6041` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:58.774752 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1bde20-190d-4b23-8f22-84e443fcc7ea admin admin] RBAC: Authorizing `identity:delete_user(user_id=dfff736ef7ea45a6a1dd3715f23c6041)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:58.776703 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1bde20-190d-4b23-8f22-84e443fcc7ea admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:58.830671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-5f1bde20-190d-4b23-8f22-84e443fcc7ea admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dfff736ef7ea45a6a1dd3715f23c6041'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:58.839465 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-5f1bde20-190d-4b23-8f22-84e443fcc7ea admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dfff736ef7ea45a6a1dd3715f23c6041'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:58.840095 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-5f1bde20-190d-4b23-8f22-84e443fcc7ea admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dfff736ef7ea45a6a1dd3715f23c6041'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:58.847970 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-5f1bde20-190d-4b23-8f22-84e443fcc7ea admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dfff736ef7ea45a6a1dd3715f23c6041'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:58.861177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-5f1bde20-190d-4b23-8f22-84e443fcc7ea admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dfff736ef7ea45a6a1dd3715f23c6041'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:58.867838 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1056/2127] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:07:58 2022] DELETE /identity/v3/users/dfff736ef7ea45a6a1dd3715f23c6041 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:07:58.872411 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfa2802-f7f6-401b-af94-64846e1fd4ae None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:58.873223 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfa2802-f7f6-401b-af94-64846e1fd4ae None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:58.873493 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfa2802-f7f6-401b-af94-64846e1fd4ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:58.894390 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-0cfa2802-f7f6-401b-af94-64846e1fd4ae None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:07:58.895560 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-0cfa2802-f7f6-401b-af94-64846e1fd4ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:58.957513 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1072/2128] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:07:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:07:58.967692 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1bde20-190d-4b23-8f22-84e443fcc7ea admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:58.969139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-5f1bde20-190d-4b23-8f22-84e443fcc7ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:58.974558 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:07:59.027780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1bde20-190d-4b23-8f22-84e443fcc7ea admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:59.031681 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f5d49e-f9ad-4bfa-bf0a-6f09c5019a51 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:59.032290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f5d49e-f9ad-4bfa-bf0a-6f09c5019a51 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:59.032583 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f5d49e-f9ad-4bfa-bf0a-6f09c5019a51 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:59.032845 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f5d49e-f9ad-4bfa-bf0a-6f09c5019a51 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:59.042056 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f5d49e-f9ad-4bfa-bf0a-6f09c5019a51 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:59.043499 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f5d49e-f9ad-4bfa-bf0a-6f09c5019a51 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:59.053134 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1057/2129] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:07:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:07:59.331429 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfa2802-f7f6-401b-af94-64846e1fd4ae None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:59.333027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-0cfa2802-f7f6-401b-af94-64846e1fd4ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:59.347543 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfa2802-f7f6-401b-af94-64846e1fd4ae None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:59.352568 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8749e16d-9975-4ab5-a71d-01f558b152b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:59.353201 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8749e16d-9975-4ab5-a71d-01f558b152b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:59.354046 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8749e16d-9975-4ab5-a71d-01f558b152b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:59.354330 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8749e16d-9975-4ab5-a71d-01f558b152b9 admin admin] PATH_INFO: `/v3/projects/ed21a370b0bb43698b8aa11334ceaad9` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:59.355758 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8749e16d-9975-4ab5-a71d-01f558b152b9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ed21a370b0bb43698b8aa11334ceaad9)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:59.357682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8749e16d-9975-4ab5-a71d-01f558b152b9 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:59.403549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:07:59.411005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8749e16d-9975-4ab5-a71d-01f558b152b9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ed21a370b0bb43698b8aa11334ceaad9', 'request_id': 'req-8749e16d-9975-4ab5-a71d-01f558b152b9'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:59.420039 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8749e16d-9975-4ab5-a71d-01f558b152b9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ed21a370b0bb43698b8aa11334ceaad9', 'request_id': 'req-8749e16d-9975-4ab5-a71d-01f558b152b9'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:59.426075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8749e16d-9975-4ab5-a71d-01f558b152b9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ed21a370b0bb43698b8aa11334ceaad9', 'request_id': 'req-8749e16d-9975-4ab5-a71d-01f558b152b9'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:59.437687 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8749e16d-9975-4ab5-a71d-01f558b152b9 admin admin] The token cache is being invalidate because project ed21a370b0bb43698b8aa11334ceaad9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:07:59.437996 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8749e16d-9975-4ab5-a71d-01f558b152b9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:59.438627 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8749e16d-9975-4ab5-a71d-01f558b152b9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:59.440194 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1073/2130] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:07:59 2022] DELETE /identity/v3/projects/ed21a370b0bb43698b8aa11334ceaad9 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:07:59.444164 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f5d49e-f9ad-4bfa-bf0a-6f09c5019a51 service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:59.453556 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f5d49e-f9ad-4bfa-bf0a-6f09c5019a51 service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:59.512676 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8410cb81-a344-4016-8218-9de2023154ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:59.513437 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8410cb81-a344-4016-8218-9de2023154ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:59.513713 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8410cb81-a344-4016-8218-9de2023154ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:59.513975 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8410cb81-a344-4016-8218-9de2023154ac admin admin] PATH_INFO: `/v3/users/81ed6d85c63747949353c1c8c75ffec4` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:59.515080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8410cb81-a344-4016-8218-9de2023154ac admin admin] RBAC: Authorizing `identity:delete_user(user_id=81ed6d85c63747949353c1c8c75ffec4)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:59.517172 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8410cb81-a344-4016-8218-9de2023154ac admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:07:59.566929 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-8410cb81-a344-4016-8218-9de2023154ac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '81ed6d85c63747949353c1c8c75ffec4'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:59.574911 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-8410cb81-a344-4016-8218-9de2023154ac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '81ed6d85c63747949353c1c8c75ffec4'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:59.575520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-8410cb81-a344-4016-8218-9de2023154ac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '81ed6d85c63747949353c1c8c75ffec4'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:59.582396 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-8410cb81-a344-4016-8218-9de2023154ac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '81ed6d85c63747949353c1c8c75ffec4'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:59.594231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-8410cb81-a344-4016-8218-9de2023154ac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '81ed6d85c63747949353c1c8c75ffec4'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:07:59.600100 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1058/2131] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:07:59 2022] DELETE /identity/v3/users/81ed6d85c63747949353c1c8c75ffec4 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:07:59.880165 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8749e16d-9975-4ab5-a71d-01f558b152b9 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:07:59.881688 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-8749e16d-9975-4ab5-a71d-01f558b152b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:07:59.899549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8749e16d-9975-4ab5-a71d-01f558b152b9 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:07:59.961163 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e0a5ed-a87e-4301-98f2-23ef892d3e2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:07:59.961797 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e0a5ed-a87e-4301-98f2-23ef892d3e2a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:07:59.962074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e0a5ed-a87e-4301-98f2-23ef892d3e2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:07:59.962671 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e0a5ed-a87e-4301-98f2-23ef892d3e2a admin admin] PATH_INFO: `/v3/projects/f2de7c6c60584571a3a57f1678a24e64` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:07:59.963879 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e0a5ed-a87e-4301-98f2-23ef892d3e2a admin admin] RBAC: Authorizing `identity:delete_project(project_id=f2de7c6c60584571a3a57f1678a24e64)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:07:59.965790 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e0a5ed-a87e-4301-98f2-23ef892d3e2a admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:00.013738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-f6e0a5ed-a87e-4301-98f2-23ef892d3e2a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f2de7c6c60584571a3a57f1678a24e64', 'request_id': 'req-f6e0a5ed-a87e-4301-98f2-23ef892d3e2a'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:00.022749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-f6e0a5ed-a87e-4301-98f2-23ef892d3e2a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f2de7c6c60584571a3a57f1678a24e64', 'request_id': 'req-f6e0a5ed-a87e-4301-98f2-23ef892d3e2a'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:00.028568 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-f6e0a5ed-a87e-4301-98f2-23ef892d3e2a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f2de7c6c60584571a3a57f1678a24e64', 'request_id': 'req-f6e0a5ed-a87e-4301-98f2-23ef892d3e2a'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:00.039825 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-f6e0a5ed-a87e-4301-98f2-23ef892d3e2a admin admin] The token cache is being invalidate because project f2de7c6c60584571a3a57f1678a24e64 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:00.040109 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-f6e0a5ed-a87e-4301-98f2-23ef892d3e2a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:00.040735 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-f6e0a5ed-a87e-4301-98f2-23ef892d3e2a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:00.042206 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1074/2132] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:07:59 2022] DELETE /identity/v3/projects/f2de7c6c60584571a3a57f1678a24e64 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:00.048990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe47e12-eb35-4537-825b-868df1fb4e2e None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:00.049502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe47e12-eb35-4537-825b-868df1fb4e2e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:00.049502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe47e12-eb35-4537-825b-868df1fb4e2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:00.070910 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-efe47e12-eb35-4537-825b-868df1fb4e2e None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:00.072173 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-efe47e12-eb35-4537-825b-868df1fb4e2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:00.091229 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:00.134685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1059/2133] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:00.143398 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22c4c8a-0de6-4ea7-8ce6-97fac7529c85 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:00.143673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22c4c8a-0de6-4ea7-8ce6-97fac7529c85 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:00.143982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22c4c8a-0de6-4ea7-8ce6-97fac7529c85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:00.164563 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-d22c4c8a-0de6-4ea7-8ce6-97fac7529c85 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:00.165747 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-d22c4c8a-0de6-4ea7-8ce6-97fac7529c85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:00.174742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1075/2134] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:08:00.178989 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe47e12-eb35-4537-825b-868df1fb4e2e None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:00.188848 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe47e12-eb35-4537-825b-868df1fb4e2e None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:00.194301 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc32663-8260-4500-a783-4fd0dc37baa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:00.194943 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc32663-8260-4500-a783-4fd0dc37baa3 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:00.195211 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc32663-8260-4500-a783-4fd0dc37baa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:00.195513 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc32663-8260-4500-a783-4fd0dc37baa3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:00.196231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc32663-8260-4500-a783-4fd0dc37baa3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:08:00.196513 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc32663-8260-4500-a783-4fd0dc37baa3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:00.198447 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc32663-8260-4500-a783-4fd0dc37baa3 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:00.210476 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1060/2135] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:08:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 01:08:00.215253 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22c4c8a-0de6-4ea7-8ce6-97fac7529c85 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:00.224777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22c4c8a-0de6-4ea7-8ce6-97fac7529c85 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:00.229805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d43e20f-bb14-4d86-af36-9f56885bf11f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:00.230395 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d43e20f-bb14-4d86-af36-9f56885bf11f admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:00.230691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d43e20f-bb14-4d86-af36-9f56885bf11f admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:00.230955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d43e20f-bb14-4d86-af36-9f56885bf11f admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:00.231570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d43e20f-bb14-4d86-af36-9f56885bf11f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:00.233524 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d43e20f-bb14-4d86-af36-9f56885bf11f admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:00.263065 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1076/2136] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:00 2022] POST /identity/v3/projects => generated 361 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:00.269514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc32663-8260-4500-a783-4fd0dc37baa3 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:00.283380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc32663-8260-4500-a783-4fd0dc37baa3 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:00.340636 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237fc2c8-3477-4d2d-85dd-2565b2038e80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:00.341239 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237fc2c8-3477-4d2d-85dd-2565b2038e80 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:00.341538 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237fc2c8-3477-4d2d-85dd-2565b2038e80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:00.341802 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237fc2c8-3477-4d2d-85dd-2565b2038e80 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:00.342426 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237fc2c8-3477-4d2d-85dd-2565b2038e80 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:00.344394 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237fc2c8-3477-4d2d-85dd-2565b2038e80 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:00.368834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1061/2137] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:00 2022] POST /identity/v3/users => generated 281 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:08:00.373018 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d43e20f-bb14-4d86-af36-9f56885bf11f admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:00.383503 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d43e20f-bb14-4d86-af36-9f56885bf11f admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:00.388170 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464786a5-8d2d-4e9f-8b38-2e9946e2ee83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:00.388762 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464786a5-8d2d-4e9f-8b38-2e9946e2ee83 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:00.389031 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464786a5-8d2d-4e9f-8b38-2e9946e2ee83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:00.389295 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464786a5-8d2d-4e9f-8b38-2e9946e2ee83 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:00.389917 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464786a5-8d2d-4e9f-8b38-2e9946e2ee83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:00.391486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464786a5-8d2d-4e9f-8b38-2e9946e2ee83 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:00.401528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1077/2138] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:00 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:00.405485 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237fc2c8-3477-4d2d-85dd-2565b2038e80 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:00.416311 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237fc2c8-3477-4d2d-85dd-2565b2038e80 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:00.421538 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d302a8d-dd79-4df5-baa5-aa4d0bacc01b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:00.422132 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d302a8d-dd79-4df5-baa5-aa4d0bacc01b admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:00.422408 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d302a8d-dd79-4df5-baa5-aa4d0bacc01b admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:00.422679 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d302a8d-dd79-4df5-baa5-aa4d0bacc01b admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:00.423351 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d302a8d-dd79-4df5-baa5-aa4d0bacc01b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:00.425132 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d302a8d-dd79-4df5-baa5-aa4d0bacc01b admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:00.436179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1062/2139] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:00 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:00.440632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464786a5-8d2d-4e9f-8b38-2e9946e2ee83 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:00.451536 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464786a5-8d2d-4e9f-8b38-2e9946e2ee83 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:00.457686 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f7c7e7-e374-460d-b9f7-56339624b783 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:00.458327 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f7c7e7-e374-460d-b9f7-56339624b783 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:00.458599 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f7c7e7-e374-460d-b9f7-56339624b783 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:00.458868 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f7c7e7-e374-460d-b9f7-56339624b783 admin admin] PATH_INFO: `/v3/projects/dbc3995b653743c380b75cb94b07c58b/users/bda9c084a3ba4705831eb12bcfbd34c8/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:00.474471 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f7c7e7-e374-460d-b9f7-56339624b783 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dbc3995b653743c380b75cb94b07c58b, user_id=bda9c084a3ba4705831eb12bcfbd34c8, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:00.477198 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f7c7e7-e374-460d-b9f7-56339624b783 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:00.479326 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:00.494434 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1078/2140] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:00 2022] PUT /identity/v3/projects/dbc3995b653743c380b75cb94b07c58b/users/bda9c084a3ba4705831eb12bcfbd34c8/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:00.505905 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfa3d75-89c6-44fa-8f97-a1de0a28ce61 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:00.506183 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfa3d75-89c6-44fa-8f97-a1de0a28ce61 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:00.506448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfa3d75-89c6-44fa-8f97-a1de0a28ce61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:00.536452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-3bfa3d75-89c6-44fa-8f97-a1de0a28ce61 None None] MFA Rules not processed for user `bda9c084a3ba4705831eb12bcfbd34c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:00.538153 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-3bfa3d75-89c6-44fa-8f97-a1de0a28ce61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:00.612843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1063/2141] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:00 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:00.647118 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f7c7e7-e374-460d-b9f7-56339624b783 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:00.649156 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-28f7c7e7-e374-460d-b9f7-56339624b783 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:00.707203 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f7c7e7-e374-460d-b9f7-56339624b783 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:00.710713 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97d391e-4652-4f33-b3e9-22197b3fa496 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:00.711345 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97d391e-4652-4f33-b3e9-22197b3fa496 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:00.711621 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97d391e-4652-4f33-b3e9-22197b3fa496 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:00.712011 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97d391e-4652-4f33-b3e9-22197b3fa496 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:00.721104 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97d391e-4652-4f33-b3e9-22197b3fa496 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:00.722531 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97d391e-4652-4f33-b3e9-22197b3fa496 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:00.731193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1079/2142] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:00 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:00.747656 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfa3d75-89c6-44fa-8f97-a1de0a28ce61 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:00.776366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:00.820794 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfa3d75-89c6-44fa-8f97-a1de0a28ce61 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:00.825528 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce0161d-dc4e-4d4e-a55c-742152161ae5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:00.826147 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce0161d-dc4e-4d4e-a55c-742152161ae5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:00.826420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce0161d-dc4e-4d4e-a55c-742152161ae5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:00.826684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce0161d-dc4e-4d4e-a55c-742152161ae5 admin admin] PATH_INFO: `/v3/users/bda9c084a3ba4705831eb12bcfbd34c8` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:00.827885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce0161d-dc4e-4d4e-a55c-742152161ae5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bda9c084a3ba4705831eb12bcfbd34c8)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:00.829900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce0161d-dc4e-4d4e-a55c-742152161ae5 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:00.888580 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-6ce0161d-dc4e-4d4e-a55c-742152161ae5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bda9c084a3ba4705831eb12bcfbd34c8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:00.897400 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-6ce0161d-dc4e-4d4e-a55c-742152161ae5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bda9c084a3ba4705831eb12bcfbd34c8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:00.898001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-6ce0161d-dc4e-4d4e-a55c-742152161ae5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bda9c084a3ba4705831eb12bcfbd34c8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:00.904918 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-6ce0161d-dc4e-4d4e-a55c-742152161ae5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bda9c084a3ba4705831eb12bcfbd34c8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:00.917510 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-6ce0161d-dc4e-4d4e-a55c-742152161ae5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bda9c084a3ba4705831eb12bcfbd34c8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:00.924254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1064/2143] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:00 2022] DELETE /identity/v3/users/bda9c084a3ba4705831eb12bcfbd34c8 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:00.929002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5979e1a6-338f-4579-b8ad-3a0f54bad241 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:00.929527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5979e1a6-338f-4579-b8ad-3a0f54bad241 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:00.929527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5979e1a6-338f-4579-b8ad-3a0f54bad241 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:00.950741 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-5979e1a6-338f-4579-b8ad-3a0f54bad241 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:00.951983 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-5979e1a6-338f-4579-b8ad-3a0f54bad241 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:01.018921 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1080/2144] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:01.029276 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce0161d-dc4e-4d4e-a55c-742152161ae5 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:01.030828 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-6ce0161d-dc4e-4d4e-a55c-742152161ae5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:01.093215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce0161d-dc4e-4d4e-a55c-742152161ae5 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:01.097851 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a36299-475c-404e-97d8-eb8a3676845a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:01.098456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a36299-475c-404e-97d8-eb8a3676845a service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:01.098730 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a36299-475c-404e-97d8-eb8a3676845a service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:01.098996 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a36299-475c-404e-97d8-eb8a3676845a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:01.108453 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a36299-475c-404e-97d8-eb8a3676845a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:01.109903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a36299-475c-404e-97d8-eb8a3676845a service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:01.119237 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1065/2145] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:01.392907 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5979e1a6-338f-4579-b8ad-3a0f54bad241 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:01.394499 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-5979e1a6-338f-4579-b8ad-3a0f54bad241 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:01.406897 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5979e1a6-338f-4579-b8ad-3a0f54bad241 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:01.412077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326af4fc-0ea5-4a89-9545-8b1ba1e4816d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:01.412742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326af4fc-0ea5-4a89-9545-8b1ba1e4816d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:01.413026 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326af4fc-0ea5-4a89-9545-8b1ba1e4816d admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:01.413294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326af4fc-0ea5-4a89-9545-8b1ba1e4816d admin admin] PATH_INFO: `/v3/projects/dbc3995b653743c380b75cb94b07c58b` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:01.414379 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326af4fc-0ea5-4a89-9545-8b1ba1e4816d admin admin] RBAC: Authorizing `identity:delete_project(project_id=dbc3995b653743c380b75cb94b07c58b)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:01.416611 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326af4fc-0ea5-4a89-9545-8b1ba1e4816d admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:01.468883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-326af4fc-0ea5-4a89-9545-8b1ba1e4816d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dbc3995b653743c380b75cb94b07c58b', 'request_id': 'req-326af4fc-0ea5-4a89-9545-8b1ba1e4816d'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:01.479343 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-326af4fc-0ea5-4a89-9545-8b1ba1e4816d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dbc3995b653743c380b75cb94b07c58b', 'request_id': 'req-326af4fc-0ea5-4a89-9545-8b1ba1e4816d'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:01.485534 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-326af4fc-0ea5-4a89-9545-8b1ba1e4816d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dbc3995b653743c380b75cb94b07c58b', 'request_id': 'req-326af4fc-0ea5-4a89-9545-8b1ba1e4816d'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:01.496728 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-326af4fc-0ea5-4a89-9545-8b1ba1e4816d admin admin] The token cache is being invalidate because project dbc3995b653743c380b75cb94b07c58b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:01.497008 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-326af4fc-0ea5-4a89-9545-8b1ba1e4816d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:01.497709 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-326af4fc-0ea5-4a89-9545-8b1ba1e4816d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:01.499275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1081/2146] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:01 2022] DELETE /identity/v3/projects/dbc3995b653743c380b75cb94b07c58b => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:01.504594 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d945c55f-f1df-4837-80d4-64198b52ae29 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:01.504868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d945c55f-f1df-4837-80d4-64198b52ae29 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:01.505101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d945c55f-f1df-4837-80d4-64198b52ae29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:01.524844 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-d945c55f-f1df-4837-80d4-64198b52ae29 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:01.526138 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d945c55f-f1df-4837-80d4-64198b52ae29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:01.565241 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:01.588677 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1066/2147] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:01.597858 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e78db3-78ce-44ba-88f7-9de5375a6841 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:01.598136 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e78db3-78ce-44ba-88f7-9de5375a6841 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:01.598401 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e78db3-78ce-44ba-88f7-9de5375a6841 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:01.618887 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-99e78db3-78ce-44ba-88f7-9de5375a6841 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:01.620043 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-99e78db3-78ce-44ba-88f7-9de5375a6841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:01.629159 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1082/2148] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:01.634232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d945c55f-f1df-4837-80d4-64198b52ae29 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:01.649476 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d945c55f-f1df-4837-80d4-64198b52ae29 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:01.657419 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0015498-a21e-4833-b1d8-0a5c1826a4dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:01.658114 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0015498-a21e-4833-b1d8-0a5c1826a4dd admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:01.658412 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0015498-a21e-4833-b1d8-0a5c1826a4dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:01.658675 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0015498-a21e-4833-b1d8-0a5c1826a4dd admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:01.659314 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0015498-a21e-4833-b1d8-0a5c1826a4dd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:08:01.659607 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0015498-a21e-4833-b1d8-0a5c1826a4dd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:01.661219 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0015498-a21e-4833-b1d8-0a5c1826a4dd admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:01.672635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1067/2149] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:08:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 01:08:01.677161 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e78db3-78ce-44ba-88f7-9de5375a6841 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:01.686428 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e78db3-78ce-44ba-88f7-9de5375a6841 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:01.691331 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf96393-aaa3-4939-9d32-111306e7c05b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:01.691960 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf96393-aaa3-4939-9d32-111306e7c05b admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:01.692229 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf96393-aaa3-4939-9d32-111306e7c05b admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:01.692489 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf96393-aaa3-4939-9d32-111306e7c05b admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:01.693075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf96393-aaa3-4939-9d32-111306e7c05b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:01.694945 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf96393-aaa3-4939-9d32-111306e7c05b admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:01.706181 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:INSERT=2,keystone:SELECT=91,keystone:DELETE=7 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:01.724577 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1083/2150] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:01 2022] POST /identity/v3/projects => generated 375 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:01.728495 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0015498-a21e-4833-b1d8-0a5c1826a4dd admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:01.739367 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0015498-a21e-4833-b1d8-0a5c1826a4dd admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:01.797251 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05091dd8-815f-4394-9883-3dc5291f31bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:01.797865 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05091dd8-815f-4394-9883-3dc5291f31bd admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:01.798208 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05091dd8-815f-4394-9883-3dc5291f31bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:01.798475 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05091dd8-815f-4394-9883-3dc5291f31bd admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:01.799103 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05091dd8-815f-4394-9883-3dc5291f31bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:01.801050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05091dd8-815f-4394-9883-3dc5291f31bd admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:01.824570 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1068/2151] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:01 2022] POST /identity/v3/users => generated 288 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:01.828771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf96393-aaa3-4939-9d32-111306e7c05b admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:01.840100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf96393-aaa3-4939-9d32-111306e7c05b admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:01.845164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acdb254-83c8-4d43-b868-2cc4164c7a83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:01.845764 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acdb254-83c8-4d43-b868-2cc4164c7a83 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:01.846040 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acdb254-83c8-4d43-b868-2cc4164c7a83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:01.846296 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acdb254-83c8-4d43-b868-2cc4164c7a83 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:01.846924 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4acdb254-83c8-4d43-b868-2cc4164c7a83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:01.848558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4acdb254-83c8-4d43-b868-2cc4164c7a83 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:01.858489 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1084/2152] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:01 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:08:01.862853 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05091dd8-815f-4394-9883-3dc5291f31bd admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:01.873552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05091dd8-815f-4394-9883-3dc5291f31bd admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:01.879361 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670aa314-ecbe-44e6-825f-56b435e53c62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:01.880053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670aa314-ecbe-44e6-825f-56b435e53c62 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:01.880364 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670aa314-ecbe-44e6-825f-56b435e53c62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:01.880684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670aa314-ecbe-44e6-825f-56b435e53c62 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:01.881322 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670aa314-ecbe-44e6-825f-56b435e53c62 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:01.882907 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670aa314-ecbe-44e6-825f-56b435e53c62 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:01.894271 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1069/2153] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:01 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:01.898633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acdb254-83c8-4d43-b868-2cc4164c7a83 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:01.908823 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acdb254-83c8-4d43-b868-2cc4164c7a83 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:01.913359 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5abe3f1-6627-40c9-adbf-1290b8a265f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:01.914004 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5abe3f1-6627-40c9-adbf-1290b8a265f2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:01.914277 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5abe3f1-6627-40c9-adbf-1290b8a265f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:01.914907 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5abe3f1-6627-40c9-adbf-1290b8a265f2 admin admin] PATH_INFO: `/v3/projects/fe22190cc3894e1c91d8b7f3e9fdb79e/users/cc616683bf6f4a7384855d43778d2615/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:01.930282 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5abe3f1-6627-40c9-adbf-1290b8a265f2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fe22190cc3894e1c91d8b7f3e9fdb79e, user_id=cc616683bf6f4a7384855d43778d2615, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:01.933014 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5abe3f1-6627-40c9-adbf-1290b8a265f2 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:01.948193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1085/2154] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:01 2022] PUT /identity/v3/projects/fe22190cc3894e1c91d8b7f3e9fdb79e/users/cc616683bf6f4a7384855d43778d2615/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:01.963780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926d3e5e-6d79-4714-b665-131f638db120 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:01.964063 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926d3e5e-6d79-4714-b665-131f638db120 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:01.964351 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926d3e5e-6d79-4714-b665-131f638db120 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:01.992002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-926d3e5e-6d79-4714-b665-131f638db120 None None] MFA Rules not processed for user `cc616683bf6f4a7384855d43778d2615`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:01.993202 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-926d3e5e-6d79-4714-b665-131f638db120 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:02.056971 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1070/2155] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:01 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:02.061624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5abe3f1-6627-40c9-adbf-1290b8a265f2 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:02.130532 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5abe3f1-6627-40c9-adbf-1290b8a265f2 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:02.136231 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff9f082-23de-4a8e-be67-d7c674ee1c56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:02.136835 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff9f082-23de-4a8e-be67-d7c674ee1c56 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:02.137096 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff9f082-23de-4a8e-be67-d7c674ee1c56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:02.137383 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff9f082-23de-4a8e-be67-d7c674ee1c56 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:02.138009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff9f082-23de-4a8e-be67-d7c674ee1c56 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:02.139988 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff9f082-23de-4a8e-be67-d7c674ee1c56 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:02.171219 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1086/2156] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:02 2022] POST /identity/v3/projects => generated 373 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:02.175841 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926d3e5e-6d79-4714-b665-131f638db120 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:02.187484 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926d3e5e-6d79-4714-b665-131f638db120 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:02.221377 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:02.247159 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df9ddca-a111-46b2-b1d9-e7b9163f6856 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:02.247834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df9ddca-a111-46b2-b1d9-e7b9163f6856 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:02.248144 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df9ddca-a111-46b2-b1d9-e7b9163f6856 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:02.248403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df9ddca-a111-46b2-b1d9-e7b9163f6856 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:02.249038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df9ddca-a111-46b2-b1d9-e7b9163f6856 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:02.250959 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df9ddca-a111-46b2-b1d9-e7b9163f6856 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:02.275432 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1071/2157] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:02 2022] POST /identity/v3/users => generated 293 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:08:02.279695 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff9f082-23de-4a8e-be67-d7c674ee1c56 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:02.290095 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff9f082-23de-4a8e-be67-d7c674ee1c56 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:02.294933 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7193ddbf-c32f-4b43-aba7-b9657f03eff7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:02.295527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7193ddbf-c32f-4b43-aba7-b9657f03eff7 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:02.295830 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7193ddbf-c32f-4b43-aba7-b9657f03eff7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:02.296099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7193ddbf-c32f-4b43-aba7-b9657f03eff7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:02.296734 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7193ddbf-c32f-4b43-aba7-b9657f03eff7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:02.298364 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7193ddbf-c32f-4b43-aba7-b9657f03eff7 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:02.309577 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1087/2158] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:02 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:02.314891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df9ddca-a111-46b2-b1d9-e7b9163f6856 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:02.326680 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df9ddca-a111-46b2-b1d9-e7b9163f6856 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:02.332682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781cff09-845a-4df1-9d8a-1c4ac3f2640b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:02.333631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781cff09-845a-4df1-9d8a-1c4ac3f2640b admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:02.333915 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781cff09-845a-4df1-9d8a-1c4ac3f2640b admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:02.334180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781cff09-845a-4df1-9d8a-1c4ac3f2640b admin admin] PATH_INFO: `/v3/projects/4eea9e97c0204e9da90011bc35a225a6/users/f1a97e6af8c142fbb21a08690d4f8703/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:02.353330 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-781cff09-845a-4df1-9d8a-1c4ac3f2640b admin admin] RBAC: Authorizing `identity:create_grant(project_id=4eea9e97c0204e9da90011bc35a225a6, user_id=f1a97e6af8c142fbb21a08690d4f8703, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:02.356045 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-781cff09-845a-4df1-9d8a-1c4ac3f2640b admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:02.371196 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1072/2159] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:02 2022] PUT /identity/v3/projects/4eea9e97c0204e9da90011bc35a225a6/users/f1a97e6af8c142fbb21a08690d4f8703/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:02.375310 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7193ddbf-c32f-4b43-aba7-b9657f03eff7 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:02.386654 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7193ddbf-c32f-4b43-aba7-b9657f03eff7 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:02.444612 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbee45b-46f9-4c7f-943f-e3465aeb993e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:02.445207 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbee45b-46f9-4c7f-943f-e3465aeb993e admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:02.445482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbee45b-46f9-4c7f-943f-e3465aeb993e admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:02.445750 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbee45b-46f9-4c7f-943f-e3465aeb993e admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:02.446391 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbee45b-46f9-4c7f-943f-e3465aeb993e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:02.448055 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbee45b-46f9-4c7f-943f-e3465aeb993e admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:02.458314 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1088/2160] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:02 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:02.463080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781cff09-845a-4df1-9d8a-1c4ac3f2640b admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:02.473987 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781cff09-845a-4df1-9d8a-1c4ac3f2640b admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:02.479302 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b84c60-62ab-4f8f-9f18-d5194b7841eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:02.479986 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b84c60-62ab-4f8f-9f18-d5194b7841eb admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:02.480254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b84c60-62ab-4f8f-9f18-d5194b7841eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:02.480514 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b84c60-62ab-4f8f-9f18-d5194b7841eb admin admin] PATH_INFO: `/v3/projects/4eea9e97c0204e9da90011bc35a225a6/users/f1a97e6af8c142fbb21a08690d4f8703/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:02.482547 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b84c60-62ab-4f8f-9f18-d5194b7841eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=4eea9e97c0204e9da90011bc35a225a6, user_id=f1a97e6af8c142fbb21a08690d4f8703, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:02.485202 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b84c60-62ab-4f8f-9f18-d5194b7841eb admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:02.495635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1073/2161] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:02 2022] PUT /identity/v3/projects/4eea9e97c0204e9da90011bc35a225a6/users/f1a97e6af8c142fbb21a08690d4f8703/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:02.506391 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c9f917-64bd-4686-8df0-d8783389b9f4 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:02.506678 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c9f917-64bd-4686-8df0-d8783389b9f4 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:02.506988 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c9f917-64bd-4686-8df0-d8783389b9f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:02.542979 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-b4c9f917-64bd-4686-8df0-d8783389b9f4 None None] MFA Rules not processed for user `f1a97e6af8c142fbb21a08690d4f8703`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:02.544226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-b4c9f917-64bd-4686-8df0-d8783389b9f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:02.557363 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:02.622927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1089/2162] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:02 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:02.637168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b84c60-62ab-4f8f-9f18-d5194b7841eb admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:02.638976 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-08b84c60-62ab-4f8f-9f18-d5194b7841eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:02.698140 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b84c60-62ab-4f8f-9f18-d5194b7841eb admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:02.702110 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d520f918-8025-48f5-9012-ad697bc6ff7b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:02.702698 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d520f918-8025-48f5-9012-ad697bc6ff7b service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:02.702967 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d520f918-8025-48f5-9012-ad697bc6ff7b service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:02.703230 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d520f918-8025-48f5-9012-ad697bc6ff7b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:02.758407 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d520f918-8025-48f5-9012-ad697bc6ff7b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:02.759995 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d520f918-8025-48f5-9012-ad697bc6ff7b service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:02.770026 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1074/2163] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:02 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:05.339473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c9f917-64bd-4686-8df0-d8783389b9f4 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:05.404123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c9f917-64bd-4686-8df0-d8783389b9f4 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:05.408931 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d197bd17-5c63-433a-a25c-a6035625bd5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:05.409676 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d197bd17-5c63-433a-a25c-a6035625bd5d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:05.410161 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d197bd17-5c63-433a-a25c-a6035625bd5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:05.411107 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d197bd17-5c63-433a-a25c-a6035625bd5d admin admin] PATH_INFO: `/v3/users/cc616683bf6f4a7384855d43778d2615` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:05.411847 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d197bd17-5c63-433a-a25c-a6035625bd5d admin admin] RBAC: Authorizing `identity:delete_user(user_id=cc616683bf6f4a7384855d43778d2615)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:05.414098 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d197bd17-5c63-433a-a25c-a6035625bd5d admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:05.470950 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-d197bd17-5c63-433a-a25c-a6035625bd5d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cc616683bf6f4a7384855d43778d2615'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:05.479563 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-d197bd17-5c63-433a-a25c-a6035625bd5d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cc616683bf6f4a7384855d43778d2615'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:05.480248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-d197bd17-5c63-433a-a25c-a6035625bd5d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cc616683bf6f4a7384855d43778d2615'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:05.492658 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-d197bd17-5c63-433a-a25c-a6035625bd5d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cc616683bf6f4a7384855d43778d2615'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:05.499403 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-d197bd17-5c63-433a-a25c-a6035625bd5d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cc616683bf6f4a7384855d43778d2615'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:05.505384 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1090/2164] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:05 2022] DELETE /identity/v3/users/cc616683bf6f4a7384855d43778d2615 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:05.509937 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503bcb7a-9b25-4c6f-8516-5f88748ad274 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:05.510481 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503bcb7a-9b25-4c6f-8516-5f88748ad274 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:05.510581 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503bcb7a-9b25-4c6f-8516-5f88748ad274 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:05.530558 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-503bcb7a-9b25-4c6f-8516-5f88748ad274 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:05.531839 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-503bcb7a-9b25-4c6f-8516-5f88748ad274 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:05.576366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:05.598105 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1075/2165] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:05.608447 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d197bd17-5c63-433a-a25c-a6035625bd5d admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:05.610114 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-d197bd17-5c63-433a-a25c-a6035625bd5d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:05.645872 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:05.670407 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d197bd17-5c63-433a-a25c-a6035625bd5d admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:05.674097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80a2cfc-c5b6-4960-81b7-331badde7805 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:05.674723 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80a2cfc-c5b6-4960-81b7-331badde7805 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:05.675008 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80a2cfc-c5b6-4960-81b7-331badde7805 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:05.675271 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80a2cfc-c5b6-4960-81b7-331badde7805 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:05.684206 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80a2cfc-c5b6-4960-81b7-331badde7805 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:05.685633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80a2cfc-c5b6-4960-81b7-331badde7805 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:05.697972 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1091/2166] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:05.987199 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503bcb7a-9b25-4c6f-8516-5f88748ad274 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:05.989725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-503bcb7a-9b25-4c6f-8516-5f88748ad274 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:06.011695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503bcb7a-9b25-4c6f-8516-5f88748ad274 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:06.016972 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada0087a-1a73-4ab9-a7e5-d5d8909f9e86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:06.017971 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada0087a-1a73-4ab9-a7e5-d5d8909f9e86 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:06.018248 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada0087a-1a73-4ab9-a7e5-d5d8909f9e86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:06.018513 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada0087a-1a73-4ab9-a7e5-d5d8909f9e86 admin admin] PATH_INFO: `/v3/projects/fe22190cc3894e1c91d8b7f3e9fdb79e` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:06.019645 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada0087a-1a73-4ab9-a7e5-d5d8909f9e86 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fe22190cc3894e1c91d8b7f3e9fdb79e)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:06.021572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada0087a-1a73-4ab9-a7e5-d5d8909f9e86 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:06.069314 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-ada0087a-1a73-4ab9-a7e5-d5d8909f9e86 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fe22190cc3894e1c91d8b7f3e9fdb79e', 'request_id': 'req-ada0087a-1a73-4ab9-a7e5-d5d8909f9e86'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:06.075137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-ada0087a-1a73-4ab9-a7e5-d5d8909f9e86 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fe22190cc3894e1c91d8b7f3e9fdb79e', 'request_id': 'req-ada0087a-1a73-4ab9-a7e5-d5d8909f9e86'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:06.083917 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-ada0087a-1a73-4ab9-a7e5-d5d8909f9e86 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fe22190cc3894e1c91d8b7f3e9fdb79e', 'request_id': 'req-ada0087a-1a73-4ab9-a7e5-d5d8909f9e86'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:06.096463 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-ada0087a-1a73-4ab9-a7e5-d5d8909f9e86 admin admin] The token cache is being invalidate because project fe22190cc3894e1c91d8b7f3e9fdb79e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:06.096635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-ada0087a-1a73-4ab9-a7e5-d5d8909f9e86 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:06.097298 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-ada0087a-1a73-4ab9-a7e5-d5d8909f9e86 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:06.098802 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1076/2167] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:05 2022] DELETE /identity/v3/projects/fe22190cc3894e1c91d8b7f3e9fdb79e => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:08:06.103286 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80a2cfc-c5b6-4960-81b7-331badde7805 service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:06.113504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80a2cfc-c5b6-4960-81b7-331badde7805 service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:06.173384 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5d1fb8-09c3-43e9-9027-664bbf623eb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:06.174091 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5d1fb8-09c3-43e9-9027-664bbf623eb3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:06.174391 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5d1fb8-09c3-43e9-9027-664bbf623eb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:06.174663 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5d1fb8-09c3-43e9-9027-664bbf623eb3 admin admin] PATH_INFO: `/v3/users/f1a97e6af8c142fbb21a08690d4f8703` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:06.175846 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5d1fb8-09c3-43e9-9027-664bbf623eb3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f1a97e6af8c142fbb21a08690d4f8703)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:06.177812 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5d1fb8-09c3-43e9-9027-664bbf623eb3 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:06.226952 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-de5d1fb8-09c3-43e9-9027-664bbf623eb3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f1a97e6af8c142fbb21a08690d4f8703'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:06.235895 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-de5d1fb8-09c3-43e9-9027-664bbf623eb3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f1a97e6af8c142fbb21a08690d4f8703'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:06.236666 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-de5d1fb8-09c3-43e9-9027-664bbf623eb3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f1a97e6af8c142fbb21a08690d4f8703'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:06.249454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-de5d1fb8-09c3-43e9-9027-664bbf623eb3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f1a97e6af8c142fbb21a08690d4f8703'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:06.256658 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-de5d1fb8-09c3-43e9-9027-664bbf623eb3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f1a97e6af8c142fbb21a08690d4f8703'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:06.262564 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1092/2168] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:06 2022] DELETE /identity/v3/users/f1a97e6af8c142fbb21a08690d4f8703 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:06.549643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada0087a-1a73-4ab9-a7e5-d5d8909f9e86 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:06.551162 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-ada0087a-1a73-4ab9-a7e5-d5d8909f9e86 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:06.564460 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada0087a-1a73-4ab9-a7e5-d5d8909f9e86 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:06.635673 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef9efcc-7551-4516-821c-7edafa298b99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:06.636402 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef9efcc-7551-4516-821c-7edafa298b99 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:06.636739 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef9efcc-7551-4516-821c-7edafa298b99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:06.637083 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef9efcc-7551-4516-821c-7edafa298b99 admin admin] PATH_INFO: `/v3/projects/4eea9e97c0204e9da90011bc35a225a6` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:06.638296 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef9efcc-7551-4516-821c-7edafa298b99 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4eea9e97c0204e9da90011bc35a225a6)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:06.640378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef9efcc-7551-4516-821c-7edafa298b99 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:06.687746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-0ef9efcc-7551-4516-821c-7edafa298b99 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4eea9e97c0204e9da90011bc35a225a6', 'request_id': 'req-0ef9efcc-7551-4516-821c-7edafa298b99'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:06.693704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-0ef9efcc-7551-4516-821c-7edafa298b99 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4eea9e97c0204e9da90011bc35a225a6', 'request_id': 'req-0ef9efcc-7551-4516-821c-7edafa298b99'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:06.702174 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-0ef9efcc-7551-4516-821c-7edafa298b99 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4eea9e97c0204e9da90011bc35a225a6', 'request_id': 'req-0ef9efcc-7551-4516-821c-7edafa298b99'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:06.710065 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:06.715643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-0ef9efcc-7551-4516-821c-7edafa298b99 admin admin] The token cache is being invalidate because project 4eea9e97c0204e9da90011bc35a225a6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:06.715972 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-0ef9efcc-7551-4516-821c-7edafa298b99 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:06.716676 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-0ef9efcc-7551-4516-821c-7edafa298b99 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:06.718865 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1077/2169] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:06 2022] DELETE /identity/v3/projects/4eea9e97c0204e9da90011bc35a225a6 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:06.723592 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2611cc3c-2140-4faf-86d8-f4f1eb88c93a None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:06.724108 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2611cc3c-2140-4faf-86d8-f4f1eb88c93a None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:06.724207 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2611cc3c-2140-4faf-86d8-f4f1eb88c93a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:06.745130 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-2611cc3c-2140-4faf-86d8-f4f1eb88c93a None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:06.746326 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-2611cc3c-2140-4faf-86d8-f4f1eb88c93a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:06.798037 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:06.813709 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1093/2170] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:06.821618 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ea5677-fd5c-4bff-8b25-7adce2af8c04 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:06.821895 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ea5677-fd5c-4bff-8b25-7adce2af8c04 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:06.822159 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ea5677-fd5c-4bff-8b25-7adce2af8c04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:06.843159 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-76ea5677-fd5c-4bff-8b25-7adce2af8c04 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:06.844416 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-76ea5677-fd5c-4bff-8b25-7adce2af8c04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:06.852547 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1078/2171] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:06.856946 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2611cc3c-2140-4faf-86d8-f4f1eb88c93a None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:06.867000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2611cc3c-2140-4faf-86d8-f4f1eb88c93a None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:06.871682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ae4f99-824f-44a2-87c4-5ed6866b1a92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:06.872312 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ae4f99-824f-44a2-87c4-5ed6866b1a92 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:06.872581 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ae4f99-824f-44a2-87c4-5ed6866b1a92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:06.872842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ae4f99-824f-44a2-87c4-5ed6866b1a92 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:06.873475 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ae4f99-824f-44a2-87c4-5ed6866b1a92 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:08:06.873766 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ae4f99-824f-44a2-87c4-5ed6866b1a92 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:06.875368 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ae4f99-824f-44a2-87c4-5ed6866b1a92 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:06.887817 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1094/2172] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:08:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 01:08:06.892043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ea5677-fd5c-4bff-8b25-7adce2af8c04 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:06.901151 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ea5677-fd5c-4bff-8b25-7adce2af8c04 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:06.906193 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b88e21-2165-40a9-b742-08af4a2ddfc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:06.906837 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b88e21-2165-40a9-b742-08af4a2ddfc1 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:06.907115 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b88e21-2165-40a9-b742-08af4a2ddfc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:06.907385 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b88e21-2165-40a9-b742-08af4a2ddfc1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:06.908163 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b88e21-2165-40a9-b742-08af4a2ddfc1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:06.910177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b88e21-2165-40a9-b742-08af4a2ddfc1 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:06.925506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ae4f99-824f-44a2-87c4-5ed6866b1a92 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:06.928592 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-b5ae4f99-824f-44a2-87c4-5ed6866b1a92 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:06.945014 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1079/2173] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:06 2022] POST /identity/v3/projects => generated 377 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:06.946421 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b88e21-2165-40a9-b742-08af4a2ddfc1 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:07.012759 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ae4f99-824f-44a2-87c4-5ed6866b1a92 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:07.015616 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b88e21-2165-40a9-b742-08af4a2ddfc1 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:07.020615 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded349de-e94a-4242-9367-4bc87d724bf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:07.021224 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded349de-e94a-4242-9367-4bc87d724bf7 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:07.021499 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded349de-e94a-4242-9367-4bc87d724bf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:07.021770 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded349de-e94a-4242-9367-4bc87d724bf7 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:07.022399 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded349de-e94a-4242-9367-4bc87d724bf7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:07.024365 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded349de-e94a-4242-9367-4bc87d724bf7 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:07.048124 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1080/2174] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:06 2022] POST /identity/v3/users => generated 289 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:08:07.052311 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded349de-e94a-4242-9367-4bc87d724bf7 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:07.063357 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded349de-e94a-4242-9367-4bc87d724bf7 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:07.069053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c6d186-1c02-47de-b95e-c0d5b13f077e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:07.069637 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c6d186-1c02-47de-b95e-c0d5b13f077e admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:07.069908 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c6d186-1c02-47de-b95e-c0d5b13f077e admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:07.070174 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c6d186-1c02-47de-b95e-c0d5b13f077e admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:07.070814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c6d186-1c02-47de-b95e-c0d5b13f077e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:07.072431 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c6d186-1c02-47de-b95e-c0d5b13f077e admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:07.076656 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80f2c70-051d-4274-bbca-9eaed5c57dfb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:07.077263 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80f2c70-051d-4274-bbca-9eaed5c57dfb service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:07.077543 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80f2c70-051d-4274-bbca-9eaed5c57dfb service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:07.077811 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80f2c70-051d-4274-bbca-9eaed5c57dfb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:07.079568 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e80f2c70-051d-4274-bbca-9eaed5c57dfb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:07.085458 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1081/2175] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:07 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:08:07.087632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c6d186-1c02-47de-b95e-c0d5b13f077e admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:07.092310 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80f2c70-051d-4274-bbca-9eaed5c57dfb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:07.094134 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80f2c70-051d-4274-bbca-9eaed5c57dfb service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:07.097359 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c6d186-1c02-47de-b95e-c0d5b13f077e admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:07.102522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51de56e6-4708-4f92-83e4-9850c67e4fec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:07.103102 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51de56e6-4708-4f92-83e4-9850c67e4fec admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:07.103372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51de56e6-4708-4f92-83e4-9850c67e4fec admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:07.103637 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51de56e6-4708-4f92-83e4-9850c67e4fec admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:07.104294 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51de56e6-4708-4f92-83e4-9850c67e4fec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:07.104875 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1095/2176] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:06 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:07.105848 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51de56e6-4708-4f92-83e4-9850c67e4fec admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:07.116918 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1082/2177] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:07.120946 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80f2c70-051d-4274-bbca-9eaed5c57dfb service nova] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:07.131435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80f2c70-051d-4274-bbca-9eaed5c57dfb service nova] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:07.136178 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ac6717-f3fb-49c9-8f5c-d257b9106753 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:07.136828 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ac6717-f3fb-49c9-8f5c-d257b9106753 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:07.137102 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ac6717-f3fb-49c9-8f5c-d257b9106753 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:07.137373 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ac6717-f3fb-49c9-8f5c-d257b9106753 admin admin] PATH_INFO: `/v3/projects/87bcdc77b34f47ef8f6cff47a2c1ed66/users/e0a564de3ee240e4b916fddb87f44f44/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:07.152816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ac6717-f3fb-49c9-8f5c-d257b9106753 admin admin] RBAC: Authorizing `identity:create_grant(project_id=87bcdc77b34f47ef8f6cff47a2c1ed66, user_id=e0a564de3ee240e4b916fddb87f44f44, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:07.155458 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ac6717-f3fb-49c9-8f5c-d257b9106753 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:07.171402 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1096/2178] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:07 2022] PUT /identity/v3/projects/87bcdc77b34f47ef8f6cff47a2c1ed66/users/e0a564de3ee240e4b916fddb87f44f44/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:08:07.182902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555b7c6c-be1f-42f7-8992-0ee33db235ee None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:07.183280 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555b7c6c-be1f-42f7-8992-0ee33db235ee None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:07.183695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555b7c6c-be1f-42f7-8992-0ee33db235ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:07.216464 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-555b7c6c-be1f-42f7-8992-0ee33db235ee None None] MFA Rules not processed for user `e0a564de3ee240e4b916fddb87f44f44`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:07.217664 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-555b7c6c-be1f-42f7-8992-0ee33db235ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:07.287785 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1083/2179] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:07 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:07.292651 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ac6717-f3fb-49c9-8f5c-d257b9106753 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:07.336512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:07.358330 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ac6717-f3fb-49c9-8f5c-d257b9106753 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:07.364338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a2c679-2a9f-4551-b512-49857cb64375 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:07.365000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a2c679-2a9f-4551-b512-49857cb64375 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:07.365272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a2c679-2a9f-4551-b512-49857cb64375 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:07.365537 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a2c679-2a9f-4551-b512-49857cb64375 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:07.366174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a2c679-2a9f-4551-b512-49857cb64375 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:07.368149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a2c679-2a9f-4551-b512-49857cb64375 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:07.397448 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1097/2180] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:07 2022] POST /identity/v3/projects => generated 373 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:07.401832 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555b7c6c-be1f-42f7-8992-0ee33db235ee None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:07.412924 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555b7c6c-be1f-42f7-8992-0ee33db235ee None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:07.437288 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:07.471391 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810fe27c-929c-48e4-a529-f077a80dfe69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:07.472035 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810fe27c-929c-48e4-a529-f077a80dfe69 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:07.472300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810fe27c-929c-48e4-a529-f077a80dfe69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:07.472565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810fe27c-929c-48e4-a529-f077a80dfe69 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:07.473197 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810fe27c-929c-48e4-a529-f077a80dfe69 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:07.475139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810fe27c-929c-48e4-a529-f077a80dfe69 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:07.498416 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1084/2181] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:07 2022] POST /identity/v3/users => generated 287 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:07.502678 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a2c679-2a9f-4551-b512-49857cb64375 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:07.513395 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a2c679-2a9f-4551-b512-49857cb64375 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:07.518949 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145845ca-a5e3-4b7d-9b06-b562ad79e975 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:07.519548 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145845ca-a5e3-4b7d-9b06-b562ad79e975 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:07.519883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145845ca-a5e3-4b7d-9b06-b562ad79e975 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:07.520152 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145845ca-a5e3-4b7d-9b06-b562ad79e975 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:07.520787 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-145845ca-a5e3-4b7d-9b06-b562ad79e975 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:07.522409 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-145845ca-a5e3-4b7d-9b06-b562ad79e975 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:07.533542 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1098/2182] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:07 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:07.537705 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810fe27c-929c-48e4-a529-f077a80dfe69 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:07.548105 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810fe27c-929c-48e4-a529-f077a80dfe69 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:07.553067 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7445d9d-b05b-4352-9661-ee0c93eeb88d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:07.553645 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7445d9d-b05b-4352-9661-ee0c93eeb88d admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:07.553920 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7445d9d-b05b-4352-9661-ee0c93eeb88d admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:07.554186 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7445d9d-b05b-4352-9661-ee0c93eeb88d admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:07.554822 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7445d9d-b05b-4352-9661-ee0c93eeb88d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:07.556455 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7445d9d-b05b-4352-9661-ee0c93eeb88d admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:07.566900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1085/2183] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:07.571163 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145845ca-a5e3-4b7d-9b06-b562ad79e975 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:07.581792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145845ca-a5e3-4b7d-9b06-b562ad79e975 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:07.588009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d029fba-2b33-406e-899e-fb961e68e2ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:07.588861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d029fba-2b33-406e-899e-fb961e68e2ff admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:07.589234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d029fba-2b33-406e-899e-fb961e68e2ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:07.589545 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d029fba-2b33-406e-899e-fb961e68e2ff admin admin] PATH_INFO: `/v3/projects/9f23ffb601b84201ad5efbaf9fedb088/users/cb9f13b08d674ce3b47647c276c59e94/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:07.612050 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d029fba-2b33-406e-899e-fb961e68e2ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f23ffb601b84201ad5efbaf9fedb088, user_id=cb9f13b08d674ce3b47647c276c59e94, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:07.614659 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d029fba-2b33-406e-899e-fb961e68e2ff admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:07.631412 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1099/2184] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:07 2022] PUT /identity/v3/projects/9f23ffb601b84201ad5efbaf9fedb088/users/cb9f13b08d674ce3b47647c276c59e94/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:07.642603 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f7a06f-06aa-49ed-a8da-9cd2886014b2 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:07.642889 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f7a06f-06aa-49ed-a8da-9cd2886014b2 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:07.643164 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f7a06f-06aa-49ed-a8da-9cd2886014b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:07.674248 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-16f7a06f-06aa-49ed-a8da-9cd2886014b2 None None] MFA Rules not processed for user `cb9f13b08d674ce3b47647c276c59e94`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:07.675469 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-16f7a06f-06aa-49ed-a8da-9cd2886014b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:07.742567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1086/2185] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:07 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:07.747295 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d029fba-2b33-406e-899e-fb961e68e2ff admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:07.812561 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d029fba-2b33-406e-899e-fb961e68e2ff admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:07.817891 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6379ff9f-9af2-4b09-82cf-6b2d219e56c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:07.818510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6379ff9f-9af2-4b09-82cf-6b2d219e56c1 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:07.818671 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6379ff9f-9af2-4b09-82cf-6b2d219e56c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:07.818959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6379ff9f-9af2-4b09-82cf-6b2d219e56c1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:07.819589 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6379ff9f-9af2-4b09-82cf-6b2d219e56c1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:07.821565 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6379ff9f-9af2-4b09-82cf-6b2d219e56c1 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:07.851433 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1100/2186] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:07 2022] POST /identity/v3/projects => generated 375 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:07.855743 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f7a06f-06aa-49ed-a8da-9cd2886014b2 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:07.866509 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f7a06f-06aa-49ed-a8da-9cd2886014b2 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:07.929719 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdb2fb5-1c99-4dd1-a48b-e032c269e648 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:07.930262 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdb2fb5-1c99-4dd1-a48b-e032c269e648 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:07.930539 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdb2fb5-1c99-4dd1-a48b-e032c269e648 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:07.930795 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdb2fb5-1c99-4dd1-a48b-e032c269e648 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:07.931422 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fdb2fb5-1c99-4dd1-a48b-e032c269e648 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:07.933389 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fdb2fb5-1c99-4dd1-a48b-e032c269e648 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:07.956804 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1087/2187] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:07 2022] POST /identity/v3/users => generated 294 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:08:07.960847 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6379ff9f-9af2-4b09-82cf-6b2d219e56c1 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:07.971637 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6379ff9f-9af2-4b09-82cf-6b2d219e56c1 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:07.976419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191ecb97-5414-4241-95e1-3429fe0b4696 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:07.977014 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191ecb97-5414-4241-95e1-3429fe0b4696 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:07.977279 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191ecb97-5414-4241-95e1-3429fe0b4696 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:07.977560 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191ecb97-5414-4241-95e1-3429fe0b4696 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:07.978188 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191ecb97-5414-4241-95e1-3429fe0b4696 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:07.979796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191ecb97-5414-4241-95e1-3429fe0b4696 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:07.992159 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1101/2188] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:07 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:07.996953 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdb2fb5-1c99-4dd1-a48b-e032c269e648 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:08.010165 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdb2fb5-1c99-4dd1-a48b-e032c269e648 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:08.017510 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546d2fae-4591-497b-a829-5941785f33da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:08.018145 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546d2fae-4591-497b-a829-5941785f33da admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:08.018433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546d2fae-4591-497b-a829-5941785f33da admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:08.018696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546d2fae-4591-497b-a829-5941785f33da admin admin] PATH_INFO: `/v3/projects/dd9921cb7c2244c1ac87c1ed5b90b3b6/users/49e0b6dc8bda4d53a108d2e738f49760/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:08.035329 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546d2fae-4591-497b-a829-5941785f33da admin admin] RBAC: Authorizing `identity:create_grant(project_id=dd9921cb7c2244c1ac87c1ed5b90b3b6, user_id=49e0b6dc8bda4d53a108d2e738f49760, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:08.038055 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546d2fae-4591-497b-a829-5941785f33da admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:08.047938 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:08.052891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1088/2189] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:07 2022] PUT /identity/v3/projects/dd9921cb7c2244c1ac87c1ed5b90b3b6/users/49e0b6dc8bda4d53a108d2e738f49760/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:08.057009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191ecb97-5414-4241-95e1-3429fe0b4696 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:08.067360 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191ecb97-5414-4241-95e1-3429fe0b4696 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:08.125406 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4f9733-a310-44fa-a68b-5617b1e72b99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:08.125995 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4f9733-a310-44fa-a68b-5617b1e72b99 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:08.126274 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4f9733-a310-44fa-a68b-5617b1e72b99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:08.126656 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4f9733-a310-44fa-a68b-5617b1e72b99 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:08.127285 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4f9733-a310-44fa-a68b-5617b1e72b99 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:08.129633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4f9733-a310-44fa-a68b-5617b1e72b99 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:08.134574 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:08.139833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1102/2190] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:08 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:08.144512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546d2fae-4591-497b-a829-5941785f33da admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:08.154951 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546d2fae-4591-497b-a829-5941785f33da admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:08.160129 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2677d044-0bca-47e1-92ef-f1a9d38d76db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:08.160773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2677d044-0bca-47e1-92ef-f1a9d38d76db admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:08.161047 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2677d044-0bca-47e1-92ef-f1a9d38d76db admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:08.161311 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2677d044-0bca-47e1-92ef-f1a9d38d76db admin admin] PATH_INFO: `/v3/projects/dd9921cb7c2244c1ac87c1ed5b90b3b6/users/49e0b6dc8bda4d53a108d2e738f49760/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:08.163450 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2677d044-0bca-47e1-92ef-f1a9d38d76db admin admin] RBAC: Authorizing `identity:create_grant(project_id=dd9921cb7c2244c1ac87c1ed5b90b3b6, user_id=49e0b6dc8bda4d53a108d2e738f49760, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:08.166102 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2677d044-0bca-47e1-92ef-f1a9d38d76db admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:08.177013 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1089/2191] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:08 2022] PUT /identity/v3/projects/dd9921cb7c2244c1ac87c1ed5b90b3b6/users/49e0b6dc8bda4d53a108d2e738f49760/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:08.187451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a9a9f9-c211-450f-b6f6-0675acef2a80 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:08.187624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a9a9f9-c211-450f-b6f6-0675acef2a80 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:08.187977 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a9a9f9-c211-450f-b6f6-0675acef2a80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:08.217091 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-c9a9a9f9-c211-450f-b6f6-0675acef2a80 None None] MFA Rules not processed for user `49e0b6dc8bda4d53a108d2e738f49760`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:08.218272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c9a9a9f9-c211-450f-b6f6-0675acef2a80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:08.290931 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1103/2192] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:08 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:08.302903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2677d044-0bca-47e1-92ef-f1a9d38d76db admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:08.304919 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-2677d044-0bca-47e1-92ef-f1a9d38d76db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:08.374139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2677d044-0bca-47e1-92ef-f1a9d38d76db admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:08.382022 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b1d5a2-6261-4dba-acf3-f2664020440c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:08.382663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b1d5a2-6261-4dba-acf3-f2664020440c service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:08.382934 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b1d5a2-6261-4dba-acf3-f2664020440c service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:08.383239 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b1d5a2-6261-4dba-acf3-f2664020440c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:08.445279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b1d5a2-6261-4dba-acf3-f2664020440c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:08.446623 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b1d5a2-6261-4dba-acf3-f2664020440c service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:08.455609 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1090/2193] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:08 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:09.979666 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a9a9f9-c211-450f-b6f6-0675acef2a80 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:09.981149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c9a9a9f9-c211-450f-b6f6-0675acef2a80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:09.993501 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a9a9f9-c211-450f-b6f6-0675acef2a80 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:09.997204 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9608c880-fa3c-4c11-ae07-1b2f46fb2b9b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:09.997803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9608c880-fa3c-4c11-ae07-1b2f46fb2b9b service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:09.998076 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9608c880-fa3c-4c11-ae07-1b2f46fb2b9b service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:09.998349 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9608c880-fa3c-4c11-ae07-1b2f46fb2b9b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:10.058335 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9608c880-fa3c-4c11-ae07-1b2f46fb2b9b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:10.059938 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9608c880-fa3c-4c11-ae07-1b2f46fb2b9b service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:10.071005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1104/2194] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:09 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:10.413066 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b1d5a2-6261-4dba-acf3-f2664020440c service cinder] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:10.414643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-80b1d5a2-6261-4dba-acf3-f2664020440c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:10.478499 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b1d5a2-6261-4dba-acf3-f2664020440c service cinder] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:10.483214 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd8f8a4-53ec-480c-bd1c-2c28dbbac43f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:10.483891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd8f8a4-53ec-480c-bd1c-2c28dbbac43f service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:10.484143 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd8f8a4-53ec-480c-bd1c-2c28dbbac43f service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:10.484518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd8f8a4-53ec-480c-bd1c-2c28dbbac43f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:10.487127 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-0cd8f8a4-53ec-480c-bd1c-2c28dbbac43f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:10.541205 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:10.553590 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd8f8a4-53ec-480c-bd1c-2c28dbbac43f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:10.555052 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd8f8a4-53ec-480c-bd1c-2c28dbbac43f service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:10.564887 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1091/2195] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:10 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:10.649181 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9608c880-fa3c-4c11-ae07-1b2f46fb2b9b service cinder] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:10.658733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9608c880-fa3c-4c11-ae07-1b2f46fb2b9b service cinder] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:10.662955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f099625-299e-4d33-8cb9-8c7ae0980e45 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:10.663927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f099625-299e-4d33-8cb9-8c7ae0980e45 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:10.664200 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f099625-299e-4d33-8cb9-8c7ae0980e45 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:10.664461 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f099625-299e-4d33-8cb9-8c7ae0980e45 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:10.674421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f099625-299e-4d33-8cb9-8c7ae0980e45 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:10.675971 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f099625-299e-4d33-8cb9-8c7ae0980e45 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:10.686643 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1105/2196] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:10 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:14.627009 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd8f8a4-53ec-480c-bd1c-2c28dbbac43f service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:14.693167 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd8f8a4-53ec-480c-bd1c-2c28dbbac43f service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:14.698934 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8898d749-f8d0-466b-bfeb-d8b3d0d847ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:14.699542 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8898d749-f8d0-466b-bfeb-d8b3d0d847ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:14.699842 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8898d749-f8d0-466b-bfeb-d8b3d0d847ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:14.700108 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8898d749-f8d0-466b-bfeb-d8b3d0d847ab admin admin] PATH_INFO: `/v3/users/e0a564de3ee240e4b916fddb87f44f44` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:14.701347 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8898d749-f8d0-466b-bfeb-d8b3d0d847ab admin admin] RBAC: Authorizing `identity:delete_user(user_id=e0a564de3ee240e4b916fddb87f44f44)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:14.703246 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8898d749-f8d0-466b-bfeb-d8b3d0d847ab admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:14.751446 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-8898d749-f8d0-466b-bfeb-d8b3d0d847ab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e0a564de3ee240e4b916fddb87f44f44'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:14.759349 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-8898d749-f8d0-466b-bfeb-d8b3d0d847ab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e0a564de3ee240e4b916fddb87f44f44'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:14.760028 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-8898d749-f8d0-466b-bfeb-d8b3d0d847ab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e0a564de3ee240e4b916fddb87f44f44'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:14.766682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-8898d749-f8d0-466b-bfeb-d8b3d0d847ab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e0a564de3ee240e4b916fddb87f44f44'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:14.778271 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-8898d749-f8d0-466b-bfeb-d8b3d0d847ab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e0a564de3ee240e4b916fddb87f44f44'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:14.784091 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1092/2197] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:14 2022] DELETE /identity/v3/users/e0a564de3ee240e4b916fddb87f44f44 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:14.789628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37de68c-f0fb-4b59-bb5f-6e528031f8ac None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:14.790140 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37de68c-f0fb-4b59-bb5f-6e528031f8ac None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:14.790240 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37de68c-f0fb-4b59-bb5f-6e528031f8ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:14.813257 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-b37de68c-f0fb-4b59-bb5f-6e528031f8ac None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:14.814476 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-b37de68c-f0fb-4b59-bb5f-6e528031f8ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:14.878716 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1106/2198] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:14.890376 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8898d749-f8d0-466b-bfeb-d8b3d0d847ab admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:14.892002 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-8898d749-f8d0-466b-bfeb-d8b3d0d847ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:14.950774 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8898d749-f8d0-466b-bfeb-d8b3d0d847ab admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:14.954778 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba188ac-79a9-4089-8073-e7d72be07102 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:14.955499 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba188ac-79a9-4089-8073-e7d72be07102 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:14.955791 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba188ac-79a9-4089-8073-e7d72be07102 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:14.956064 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba188ac-79a9-4089-8073-e7d72be07102 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:14.966966 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba188ac-79a9-4089-8073-e7d72be07102 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:14.968752 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba188ac-79a9-4089-8073-e7d72be07102 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:14.980504 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1093/2199] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:15.253595 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37de68c-f0fb-4b59-bb5f-6e528031f8ac None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:15.255469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-b37de68c-f0fb-4b59-bb5f-6e528031f8ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:15.270261 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37de68c-f0fb-4b59-bb5f-6e528031f8ac None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:15.277554 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292531b7-a5de-43d0-bc2b-33f462392423 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:15.278310 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292531b7-a5de-43d0-bc2b-33f462392423 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:15.278652 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292531b7-a5de-43d0-bc2b-33f462392423 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:15.279099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292531b7-a5de-43d0-bc2b-33f462392423 admin admin] PATH_INFO: `/v3/projects/87bcdc77b34f47ef8f6cff47a2c1ed66` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:15.280550 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292531b7-a5de-43d0-bc2b-33f462392423 admin admin] RBAC: Authorizing `identity:delete_project(project_id=87bcdc77b34f47ef8f6cff47a2c1ed66)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:15.282975 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292531b7-a5de-43d0-bc2b-33f462392423 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:15.290499 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:15.337160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-292531b7-a5de-43d0-bc2b-33f462392423 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '87bcdc77b34f47ef8f6cff47a2c1ed66', 'request_id': 'req-292531b7-a5de-43d0-bc2b-33f462392423'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:15.348748 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-292531b7-a5de-43d0-bc2b-33f462392423 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '87bcdc77b34f47ef8f6cff47a2c1ed66', 'request_id': 'req-292531b7-a5de-43d0-bc2b-33f462392423'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:15.355541 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-292531b7-a5de-43d0-bc2b-33f462392423 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '87bcdc77b34f47ef8f6cff47a2c1ed66', 'request_id': 'req-292531b7-a5de-43d0-bc2b-33f462392423'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:15.371324 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-292531b7-a5de-43d0-bc2b-33f462392423 admin admin] The token cache is being invalidate because project 87bcdc77b34f47ef8f6cff47a2c1ed66 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:15.371612 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-292531b7-a5de-43d0-bc2b-33f462392423 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:15.372308 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-292531b7-a5de-43d0-bc2b-33f462392423 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:15.373806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1107/2200] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:15 2022] DELETE /identity/v3/projects/87bcdc77b34f47ef8f6cff47a2c1ed66 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:15.378064 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba188ac-79a9-4089-8073-e7d72be07102 service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:15.388205 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba188ac-79a9-4089-8073-e7d72be07102 service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:15.398921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:15.450698 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13127db3-74a0-43ea-aa7c-6c520d4c71cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:15.451353 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13127db3-74a0-43ea-aa7c-6c520d4c71cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:15.451625 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13127db3-74a0-43ea-aa7c-6c520d4c71cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:15.451926 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13127db3-74a0-43ea-aa7c-6c520d4c71cc admin admin] PATH_INFO: `/v3/users/cb9f13b08d674ce3b47647c276c59e94` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:15.453075 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13127db3-74a0-43ea-aa7c-6c520d4c71cc admin admin] RBAC: Authorizing `identity:delete_user(user_id=cb9f13b08d674ce3b47647c276c59e94)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:15.454998 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13127db3-74a0-43ea-aa7c-6c520d4c71cc admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:15.505591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-13127db3-74a0-43ea-aa7c-6c520d4c71cc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cb9f13b08d674ce3b47647c276c59e94'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:15.514813 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-13127db3-74a0-43ea-aa7c-6c520d4c71cc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cb9f13b08d674ce3b47647c276c59e94'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:15.515537 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-13127db3-74a0-43ea-aa7c-6c520d4c71cc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cb9f13b08d674ce3b47647c276c59e94'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:15.523264 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-13127db3-74a0-43ea-aa7c-6c520d4c71cc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cb9f13b08d674ce3b47647c276c59e94'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:15.535913 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-13127db3-74a0-43ea-aa7c-6c520d4c71cc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cb9f13b08d674ce3b47647c276c59e94'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:15.541853 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1094/2201] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:15 2022] DELETE /identity/v3/users/cb9f13b08d674ce3b47647c276c59e94 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:08:15.827456 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292531b7-a5de-43d0-bc2b-33f462392423 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:15.829063 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-292531b7-a5de-43d0-bc2b-33f462392423 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:15.846793 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292531b7-a5de-43d0-bc2b-33f462392423 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:15.906633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b476e8f9-b9d9-4679-bb30-b684bced7aca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:15.907217 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b476e8f9-b9d9-4679-bb30-b684bced7aca admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:15.907492 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b476e8f9-b9d9-4679-bb30-b684bced7aca admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:15.907791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b476e8f9-b9d9-4679-bb30-b684bced7aca admin admin] PATH_INFO: `/v3/projects/9f23ffb601b84201ad5efbaf9fedb088` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:15.909239 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b476e8f9-b9d9-4679-bb30-b684bced7aca admin admin] RBAC: Authorizing `identity:delete_project(project_id=9f23ffb601b84201ad5efbaf9fedb088)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:15.911160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b476e8f9-b9d9-4679-bb30-b684bced7aca admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:15.963628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b476e8f9-b9d9-4679-bb30-b684bced7aca admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9f23ffb601b84201ad5efbaf9fedb088', 'request_id': 'req-b476e8f9-b9d9-4679-bb30-b684bced7aca'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:15.973754 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b476e8f9-b9d9-4679-bb30-b684bced7aca admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9f23ffb601b84201ad5efbaf9fedb088', 'request_id': 'req-b476e8f9-b9d9-4679-bb30-b684bced7aca'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:15.979712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b476e8f9-b9d9-4679-bb30-b684bced7aca admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9f23ffb601b84201ad5efbaf9fedb088', 'request_id': 'req-b476e8f9-b9d9-4679-bb30-b684bced7aca'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:15.991119 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b476e8f9-b9d9-4679-bb30-b684bced7aca admin admin] The token cache is being invalidate because project 9f23ffb601b84201ad5efbaf9fedb088 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:15.991405 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b476e8f9-b9d9-4679-bb30-b684bced7aca admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:15.992265 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b476e8f9-b9d9-4679-bb30-b684bced7aca admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:15.994446 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1108/2202] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:15 2022] DELETE /identity/v3/projects/9f23ffb601b84201ad5efbaf9fedb088 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:16.000118 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13127db3-74a0-43ea-aa7c-6c520d4c71cc admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:16.003360 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-13127db3-74a0-43ea-aa7c-6c520d4c71cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:16.016996 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13127db3-74a0-43ea-aa7c-6c520d4c71cc admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:16.075513 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3516cff-c2c2-4dad-bc17-08f8881722d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:16.076187 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3516cff-c2c2-4dad-bc17-08f8881722d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:16.076459 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3516cff-c2c2-4dad-bc17-08f8881722d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:16.076729 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3516cff-c2c2-4dad-bc17-08f8881722d8 admin admin] PATH_INFO: `/v3/users/49e0b6dc8bda4d53a108d2e738f49760` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:16.077879 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3516cff-c2c2-4dad-bc17-08f8881722d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=49e0b6dc8bda4d53a108d2e738f49760)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:16.079833 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3516cff-c2c2-4dad-bc17-08f8881722d8 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:16.120588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:16.131478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-c3516cff-c2c2-4dad-bc17-08f8881722d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '49e0b6dc8bda4d53a108d2e738f49760'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:16.141522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-c3516cff-c2c2-4dad-bc17-08f8881722d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '49e0b6dc8bda4d53a108d2e738f49760'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:16.142244 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-c3516cff-c2c2-4dad-bc17-08f8881722d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '49e0b6dc8bda4d53a108d2e738f49760'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:16.149610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-c3516cff-c2c2-4dad-bc17-08f8881722d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '49e0b6dc8bda4d53a108d2e738f49760'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:16.162381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-c3516cff-c2c2-4dad-bc17-08f8881722d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '49e0b6dc8bda4d53a108d2e738f49760'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:16.168517 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1095/2203] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:15 2022] DELETE /identity/v3/users/49e0b6dc8bda4d53a108d2e738f49760 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:16.687341 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b476e8f9-b9d9-4679-bb30-b684bced7aca admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:16.689123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-b476e8f9-b9d9-4679-bb30-b684bced7aca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:16.704496 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b476e8f9-b9d9-4679-bb30-b684bced7aca admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:16.727568 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:16.767396 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0723ce-d4ff-42cc-b743-f0ba21c1bce3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:16.768084 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0723ce-d4ff-42cc-b743-f0ba21c1bce3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:16.768361 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0723ce-d4ff-42cc-b743-f0ba21c1bce3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:16.768628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0723ce-d4ff-42cc-b743-f0ba21c1bce3 admin admin] PATH_INFO: `/v3/projects/dd9921cb7c2244c1ac87c1ed5b90b3b6` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:16.769716 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0723ce-d4ff-42cc-b743-f0ba21c1bce3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dd9921cb7c2244c1ac87c1ed5b90b3b6)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:16.771644 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0723ce-d4ff-42cc-b743-f0ba21c1bce3 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:16.821634 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8d0723ce-d4ff-42cc-b743-f0ba21c1bce3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dd9921cb7c2244c1ac87c1ed5b90b3b6', 'request_id': 'req-8d0723ce-d4ff-42cc-b743-f0ba21c1bce3'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:16.832309 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8d0723ce-d4ff-42cc-b743-f0ba21c1bce3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dd9921cb7c2244c1ac87c1ed5b90b3b6', 'request_id': 'req-8d0723ce-d4ff-42cc-b743-f0ba21c1bce3'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:16.838341 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8d0723ce-d4ff-42cc-b743-f0ba21c1bce3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dd9921cb7c2244c1ac87c1ed5b90b3b6', 'request_id': 'req-8d0723ce-d4ff-42cc-b743-f0ba21c1bce3'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:16.849637 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8d0723ce-d4ff-42cc-b743-f0ba21c1bce3 admin admin] The token cache is being invalidate because project dd9921cb7c2244c1ac87c1ed5b90b3b6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:16.849917 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8d0723ce-d4ff-42cc-b743-f0ba21c1bce3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:16.850651 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8d0723ce-d4ff-42cc-b743-f0ba21c1bce3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:16.852275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1109/2204] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:16 2022] DELETE /identity/v3/projects/dd9921cb7c2244c1ac87c1ed5b90b3b6 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:16.857491 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce38cac4-d6d9-4d25-994b-865fae952fda None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:16.857771 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce38cac4-d6d9-4d25-994b-865fae952fda None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:16.858038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce38cac4-d6d9-4d25-994b-865fae952fda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:16.878304 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-ce38cac4-d6d9-4d25-994b-865fae952fda None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:16.879494 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-ce38cac4-d6d9-4d25-994b-865fae952fda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:16.939157 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1096/2205] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:16.947311 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bade5877-68c3-4a5e-935e-2e4b2c5f82a0 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:16.947593 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bade5877-68c3-4a5e-935e-2e4b2c5f82a0 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:16.947898 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bade5877-68c3-4a5e-935e-2e4b2c5f82a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:16.970414 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-bade5877-68c3-4a5e-935e-2e4b2c5f82a0 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:16.971632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-bade5877-68c3-4a5e-935e-2e4b2c5f82a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:16.980586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1110/2206] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:16.984901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce38cac4-d6d9-4d25-994b-865fae952fda None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:16.995367 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce38cac4-d6d9-4d25-994b-865fae952fda None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:17.000722 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd83b0ad-27c3-4895-92ab-5a108e5a5a48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:17.001307 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd83b0ad-27c3-4895-92ab-5a108e5a5a48 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:17.001571 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd83b0ad-27c3-4895-92ab-5a108e5a5a48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:17.001804 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd83b0ad-27c3-4895-92ab-5a108e5a5a48 admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:17.002585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd83b0ad-27c3-4895-92ab-5a108e5a5a48 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:08:17.002870 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd83b0ad-27c3-4895-92ab-5a108e5a5a48 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:17.004478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd83b0ad-27c3-4895-92ab-5a108e5a5a48 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:17.016681 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1097/2207] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:08:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 01:08:17.021408 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bade5877-68c3-4a5e-935e-2e4b2c5f82a0 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:17.030874 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bade5877-68c3-4a5e-935e-2e4b2c5f82a0 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:17.036782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71185f77-20ce-462d-b7cf-d2efa56a504b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:17.037379 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71185f77-20ce-462d-b7cf-d2efa56a504b admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:17.037657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71185f77-20ce-462d-b7cf-d2efa56a504b admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:17.037923 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71185f77-20ce-462d-b7cf-d2efa56a504b admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:17.038549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71185f77-20ce-462d-b7cf-d2efa56a504b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:17.040514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71185f77-20ce-462d-b7cf-d2efa56a504b admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:17.071433 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1111/2208] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:17 2022] POST /identity/v3/projects => generated 385 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:17.075934 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd83b0ad-27c3-4895-92ab-5a108e5a5a48 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:17.086041 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd83b0ad-27c3-4895-92ab-5a108e5a5a48 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:17.142239 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a13303-03b5-48aa-b85e-b4d11047c396 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:17.142896 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a13303-03b5-48aa-b85e-b4d11047c396 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:17.143166 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a13303-03b5-48aa-b85e-b4d11047c396 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:17.143426 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a13303-03b5-48aa-b85e-b4d11047c396 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:17.144056 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a13303-03b5-48aa-b85e-b4d11047c396 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:17.145960 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a13303-03b5-48aa-b85e-b4d11047c396 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:17.170467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1098/2209] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:17 2022] POST /identity/v3/users => generated 293 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:17.175448 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71185f77-20ce-462d-b7cf-d2efa56a504b admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:17.186215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71185f77-20ce-462d-b7cf-d2efa56a504b admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:17.191867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26d61dc-74ef-4097-b84e-54718ad3d2d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:17.192417 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26d61dc-74ef-4097-b84e-54718ad3d2d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:17.192691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26d61dc-74ef-4097-b84e-54718ad3d2d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:17.192984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26d61dc-74ef-4097-b84e-54718ad3d2d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:17.193651 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26d61dc-74ef-4097-b84e-54718ad3d2d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:17.195347 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26d61dc-74ef-4097-b84e-54718ad3d2d4 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:17.206411 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1112/2210] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:17 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:08:17.210611 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a13303-03b5-48aa-b85e-b4d11047c396 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:17.221456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a13303-03b5-48aa-b85e-b4d11047c396 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:17.227413 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1751bc46-1e34-42e1-b30f-2bd1362afaa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:17.228044 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1751bc46-1e34-42e1-b30f-2bd1362afaa3 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:17.228316 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1751bc46-1e34-42e1-b30f-2bd1362afaa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:17.228577 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1751bc46-1e34-42e1-b30f-2bd1362afaa3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:17.229208 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1751bc46-1e34-42e1-b30f-2bd1362afaa3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:17.230804 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1751bc46-1e34-42e1-b30f-2bd1362afaa3 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:17.241858 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1099/2211] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:17 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:17.247210 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26d61dc-74ef-4097-b84e-54718ad3d2d4 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:17.258129 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26d61dc-74ef-4097-b84e-54718ad3d2d4 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:17.264028 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de0885a-11e0-4e63-9e16-186629d038e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:17.264704 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de0885a-11e0-4e63-9e16-186629d038e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:17.264978 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de0885a-11e0-4e63-9e16-186629d038e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:17.265247 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de0885a-11e0-4e63-9e16-186629d038e7 admin admin] PATH_INFO: `/v3/projects/1ca77f31e2244ba1b13d57cbcf797c20/users/bcbf7700a36e45da8642b44b01c65a4b/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:17.280415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de0885a-11e0-4e63-9e16-186629d038e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ca77f31e2244ba1b13d57cbcf797c20, user_id=bcbf7700a36e45da8642b44b01c65a4b, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:17.283053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de0885a-11e0-4e63-9e16-186629d038e7 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:17.299855 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1113/2212] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:17 2022] PUT /identity/v3/projects/1ca77f31e2244ba1b13d57cbcf797c20/users/bcbf7700a36e45da8642b44b01c65a4b/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:17.311157 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c863c2ae-4481-4254-bca9-a36884b07dfa None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:17.311446 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c863c2ae-4481-4254-bca9-a36884b07dfa None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:17.311731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c863c2ae-4481-4254-bca9-a36884b07dfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:17.341281 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-c863c2ae-4481-4254-bca9-a36884b07dfa None None] MFA Rules not processed for user `bcbf7700a36e45da8642b44b01c65a4b`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:17.342523 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-c863c2ae-4481-4254-bca9-a36884b07dfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:17.347843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:17.411905 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1100/2213] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:17 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:17.417362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de0885a-11e0-4e63-9e16-186629d038e7 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:17.488138 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:17.501214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de0885a-11e0-4e63-9e16-186629d038e7 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:17.507187 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cb6204-9b80-4a8e-afdf-ae6b39ce203a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:17.507991 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cb6204-9b80-4a8e-afdf-ae6b39ce203a admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:17.508348 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cb6204-9b80-4a8e-afdf-ae6b39ce203a admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:17.508852 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cb6204-9b80-4a8e-afdf-ae6b39ce203a admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:17.509485 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6cb6204-9b80-4a8e-afdf-ae6b39ce203a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:17.511411 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6cb6204-9b80-4a8e-afdf-ae6b39ce203a admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:17.543030 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1114/2214] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:17 2022] POST /identity/v3/projects => generated 383 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:17.547124 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c863c2ae-4481-4254-bca9-a36884b07dfa None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:17.558059 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c863c2ae-4481-4254-bca9-a36884b07dfa None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:17.624328 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7907e0-a680-45df-afd7-0217996f827b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:17.625145 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7907e0-a680-45df-afd7-0217996f827b admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:17.625489 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7907e0-a680-45df-afd7-0217996f827b admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:17.625867 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7907e0-a680-45df-afd7-0217996f827b admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:17.626778 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7907e0-a680-45df-afd7-0217996f827b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:17.629179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7907e0-a680-45df-afd7-0217996f827b admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:17.653415 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1101/2215] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:17 2022] POST /identity/v3/users => generated 292 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:17.657613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cb6204-9b80-4a8e-afdf-ae6b39ce203a admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:17.668068 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cb6204-9b80-4a8e-afdf-ae6b39ce203a admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:17.672848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ceca37b-ac67-41a0-9d9c-8fc63a3a1662 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:17.673441 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ceca37b-ac67-41a0-9d9c-8fc63a3a1662 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:17.673709 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ceca37b-ac67-41a0-9d9c-8fc63a3a1662 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:17.673971 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ceca37b-ac67-41a0-9d9c-8fc63a3a1662 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:17.674735 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ceca37b-ac67-41a0-9d9c-8fc63a3a1662 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:17.676317 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ceca37b-ac67-41a0-9d9c-8fc63a3a1662 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:17.687322 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1115/2216] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:17 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:17.692392 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7907e0-a680-45df-afd7-0217996f827b admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:17.707557 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7907e0-a680-45df-afd7-0217996f827b admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:17.713709 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e126a9-7282-4a9b-a7bb-d5959767550c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:17.714313 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e126a9-7282-4a9b-a7bb-d5959767550c admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:17.714585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e126a9-7282-4a9b-a7bb-d5959767550c admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:17.714875 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e126a9-7282-4a9b-a7bb-d5959767550c admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:17.715510 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e126a9-7282-4a9b-a7bb-d5959767550c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:17.717249 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e126a9-7282-4a9b-a7bb-d5959767550c admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:17.727742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1102/2217] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:17 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:17.731983 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ceca37b-ac67-41a0-9d9c-8fc63a3a1662 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:17.742307 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ceca37b-ac67-41a0-9d9c-8fc63a3a1662 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:17.747404 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8296d2c6-d02e-4831-bb03-2960a7e95763 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:17.748065 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8296d2c6-d02e-4831-bb03-2960a7e95763 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:17.748479 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8296d2c6-d02e-4831-bb03-2960a7e95763 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:17.748898 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8296d2c6-d02e-4831-bb03-2960a7e95763 admin admin] PATH_INFO: `/v3/projects/63f1234bf985406d9886d95e81a4770e/users/7f711122516d4ca6a3651879d7b9fff2/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:17.764170 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8296d2c6-d02e-4831-bb03-2960a7e95763 admin admin] RBAC: Authorizing `identity:create_grant(project_id=63f1234bf985406d9886d95e81a4770e, user_id=7f711122516d4ca6a3651879d7b9fff2, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:17.766840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8296d2c6-d02e-4831-bb03-2960a7e95763 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:17.782714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1116/2218] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:17 2022] PUT /identity/v3/projects/63f1234bf985406d9886d95e81a4770e/users/7f711122516d4ca6a3651879d7b9fff2/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:17.793717 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698ad183-79ec-4d34-a386-519bac4d548b None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:17.794000 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698ad183-79ec-4d34-a386-519bac4d548b None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:17.794276 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698ad183-79ec-4d34-a386-519bac4d548b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:17.822951 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-698ad183-79ec-4d34-a386-519bac4d548b None None] MFA Rules not processed for user `7f711122516d4ca6a3651879d7b9fff2`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:17.824182 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-698ad183-79ec-4d34-a386-519bac4d548b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:17.891859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1103/2219] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:17 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:17.896756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8296d2c6-d02e-4831-bb03-2960a7e95763 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:17.961016 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8296d2c6-d02e-4831-bb03-2960a7e95763 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:17.966382 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fe9908-281c-4802-b31a-7bae23cc351b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:17.966978 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fe9908-281c-4802-b31a-7bae23cc351b admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:17.967248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fe9908-281c-4802-b31a-7bae23cc351b admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:17.967584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fe9908-281c-4802-b31a-7bae23cc351b admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:17.968183 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fe9908-281c-4802-b31a-7bae23cc351b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:17.970085 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fe9908-281c-4802-b31a-7bae23cc351b admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:17.998857 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1117/2220] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:17 2022] POST /identity/v3/projects => generated 383 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:18.003252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698ad183-79ec-4d34-a386-519bac4d548b None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:18.014073 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698ad183-79ec-4d34-a386-519bac4d548b None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:18.026022 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:18.073492 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da391888-4a76-4f29-94e8-40afc93b99a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:18.074096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da391888-4a76-4f29-94e8-40afc93b99a9 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:18.074361 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da391888-4a76-4f29-94e8-40afc93b99a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:18.074645 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da391888-4a76-4f29-94e8-40afc93b99a9 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:18.075344 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da391888-4a76-4f29-94e8-40afc93b99a9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:18.077287 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da391888-4a76-4f29-94e8-40afc93b99a9 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:18.102381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1104/2221] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:18 2022] POST /identity/v3/users => generated 298 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:18.106551 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fe9908-281c-4802-b31a-7bae23cc351b admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:18.117275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fe9908-281c-4802-b31a-7bae23cc351b admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:18.122151 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafb6d7b-32ae-473d-9abb-c5e6b1459791 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:18.122797 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafb6d7b-32ae-473d-9abb-c5e6b1459791 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:18.123073 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafb6d7b-32ae-473d-9abb-c5e6b1459791 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:18.123338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafb6d7b-32ae-473d-9abb-c5e6b1459791 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:18.124031 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafb6d7b-32ae-473d-9abb-c5e6b1459791 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:18.125760 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafb6d7b-32ae-473d-9abb-c5e6b1459791 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:18.136740 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1118/2222] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:18 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:18.140887 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da391888-4a76-4f29-94e8-40afc93b99a9 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:18.151598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da391888-4a76-4f29-94e8-40afc93b99a9 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:18.157454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5822c088-e46e-4b00-8229-0d0fec73f177 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:18.158084 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5822c088-e46e-4b00-8229-0d0fec73f177 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:18.158520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5822c088-e46e-4b00-8229-0d0fec73f177 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:18.158760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5822c088-e46e-4b00-8229-0d0fec73f177 admin admin] PATH_INFO: `/v3/projects/18c2e00e366e496face932da216d6946/users/aa4d8660753b4f8393acb6f684a5c1b3/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:18.175549 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5822c088-e46e-4b00-8229-0d0fec73f177 admin admin] RBAC: Authorizing `identity:create_grant(project_id=18c2e00e366e496face932da216d6946, user_id=aa4d8660753b4f8393acb6f684a5c1b3, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:18.178209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5822c088-e46e-4b00-8229-0d0fec73f177 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:18.195205 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1105/2223] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:18 2022] PUT /identity/v3/projects/18c2e00e366e496face932da216d6946/users/aa4d8660753b4f8393acb6f684a5c1b3/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:18.199568 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafb6d7b-32ae-473d-9abb-c5e6b1459791 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:18.210440 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafb6d7b-32ae-473d-9abb-c5e6b1459791 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:18.268621 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbbee8a-b9d7-45c5-a4ad-21c0ec6c57fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:18.269220 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbbee8a-b9d7-45c5-a4ad-21c0ec6c57fb admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:18.269492 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbbee8a-b9d7-45c5-a4ad-21c0ec6c57fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:18.269756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbbee8a-b9d7-45c5-a4ad-21c0ec6c57fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:18.270380 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbbee8a-b9d7-45c5-a4ad-21c0ec6c57fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:18.271994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbbee8a-b9d7-45c5-a4ad-21c0ec6c57fb admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:18.284768 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1119/2224] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:18 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:18.289137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5822c088-e46e-4b00-8229-0d0fec73f177 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:18.300232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5822c088-e46e-4b00-8229-0d0fec73f177 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:18.305511 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c7485c-15f3-4194-93c9-71ddf86e45ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:18.306178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c7485c-15f3-4194-93c9-71ddf86e45ea admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:18.306468 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c7485c-15f3-4194-93c9-71ddf86e45ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:18.306730 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c7485c-15f3-4194-93c9-71ddf86e45ea admin admin] PATH_INFO: `/v3/projects/18c2e00e366e496face932da216d6946/users/aa4d8660753b4f8393acb6f684a5c1b3/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:18.308862 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c7485c-15f3-4194-93c9-71ddf86e45ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=18c2e00e366e496face932da216d6946, user_id=aa4d8660753b4f8393acb6f684a5c1b3, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:18.311473 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c7485c-15f3-4194-93c9-71ddf86e45ea admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:18.323042 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1106/2225] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:18 2022] PUT /identity/v3/projects/18c2e00e366e496face932da216d6946/users/aa4d8660753b4f8393acb6f684a5c1b3/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:18.335079 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfffca90-0d13-4b84-927e-683d2051cab9 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:18.335391 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfffca90-0d13-4b84-927e-683d2051cab9 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:18.335805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfffca90-0d13-4b84-927e-683d2051cab9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:18.370629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:18.381994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-cfffca90-0d13-4b84-927e-683d2051cab9 None None] MFA Rules not processed for user `aa4d8660753b4f8393acb6f684a5c1b3`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:18.384108 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-cfffca90-0d13-4b84-927e-683d2051cab9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:18.458145 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1120/2226] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:18 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:18.476529 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c7485c-15f3-4194-93c9-71ddf86e45ea admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:18.479412 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-f0c7485c-15f3-4194-93c9-71ddf86e45ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:18.544101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c7485c-15f3-4194-93c9-71ddf86e45ea admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:18.548413 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78d672e-76e8-4e5c-9b25-15ddd81d7e88 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:18.549093 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78d672e-76e8-4e5c-9b25-15ddd81d7e88 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:18.549362 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78d672e-76e8-4e5c-9b25-15ddd81d7e88 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:18.549658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78d672e-76e8-4e5c-9b25-15ddd81d7e88 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:18.609267 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78d672e-76e8-4e5c-9b25-15ddd81d7e88 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:18.610701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78d672e-76e8-4e5c-9b25-15ddd81d7e88 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:18.619860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1107/2227] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:18 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:20.216382 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfffca90-0d13-4b84-927e-683d2051cab9 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:20.226773 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfffca90-0d13-4b84-927e-683d2051cab9 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:20.231278 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a725667-c461-49e9-8a70-cc30c3e9c5ea service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:20.231909 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a725667-c461-49e9-8a70-cc30c3e9c5ea service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:20.232178 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a725667-c461-49e9-8a70-cc30c3e9c5ea service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:20.232438 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a725667-c461-49e9-8a70-cc30c3e9c5ea service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:20.292719 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a725667-c461-49e9-8a70-cc30c3e9c5ea service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:20.294159 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a725667-c461-49e9-8a70-cc30c3e9c5ea service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:20.303546 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1121/2228] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:20 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:20.891600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78d672e-76e8-4e5c-9b25-15ddd81d7e88 service cinder] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:20.900830 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78d672e-76e8-4e5c-9b25-15ddd81d7e88 service cinder] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:20.905146 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1ec30b-8cdd-4f4a-b261-36dfbd25dfd9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:20.905763 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1ec30b-8cdd-4f4a-b261-36dfbd25dfd9 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:20.906032 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1ec30b-8cdd-4f4a-b261-36dfbd25dfd9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:20.906328 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1ec30b-8cdd-4f4a-b261-36dfbd25dfd9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:20.915956 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1ec30b-8cdd-4f4a-b261-36dfbd25dfd9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:20.917384 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1ec30b-8cdd-4f4a-b261-36dfbd25dfd9 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:20.927101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1108/2229] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:20 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:24.878979 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a725667-c461-49e9-8a70-cc30c3e9c5ea service cinder] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:24.945542 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a725667-c461-49e9-8a70-cc30c3e9c5ea service cinder] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:24.950087 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd941f4-f5b8-475c-99b8-b4bb44d8b964 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:24.950766 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd941f4-f5b8-475c-99b8-b4bb44d8b964 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:24.950919 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd941f4-f5b8-475c-99b8-b4bb44d8b964 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:24.951219 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd941f4-f5b8-475c-99b8-b4bb44d8b964 admin admin] PATH_INFO: `/v3/users/bcbf7700a36e45da8642b44b01c65a4b` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:24.952336 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd941f4-f5b8-475c-99b8-b4bb44d8b964 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bcbf7700a36e45da8642b44b01c65a4b)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:24.954234 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd941f4-f5b8-475c-99b8-b4bb44d8b964 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:24.997537 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:25.006469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-6cd941f4-f5b8-475c-99b8-b4bb44d8b964 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bcbf7700a36e45da8642b44b01c65a4b'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:25.015679 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-6cd941f4-f5b8-475c-99b8-b4bb44d8b964 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bcbf7700a36e45da8642b44b01c65a4b'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:25.016313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-6cd941f4-f5b8-475c-99b8-b4bb44d8b964 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bcbf7700a36e45da8642b44b01c65a4b'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:25.028107 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-6cd941f4-f5b8-475c-99b8-b4bb44d8b964 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bcbf7700a36e45da8642b44b01c65a4b'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:25.034870 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-6cd941f4-f5b8-475c-99b8-b4bb44d8b964 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bcbf7700a36e45da8642b44b01c65a4b'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:25.041232 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1122/2230] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:24 2022] DELETE /identity/v3/users/bcbf7700a36e45da8642b44b01c65a4b => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:25.046039 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fd1628-7f3b-4281-9911-f43a8230bca9 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:25.046531 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fd1628-7f3b-4281-9911-f43a8230bca9 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:25.046647 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fd1628-7f3b-4281-9911-f43a8230bca9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:25.066228 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-11fd1628-7f3b-4281-9911-f43a8230bca9 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:25.067440 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-11fd1628-7f3b-4281-9911-f43a8230bca9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:25.072145 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:25.131674 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1109/2231] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:25.143432 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd941f4-f5b8-475c-99b8-b4bb44d8b964 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:25.144964 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-6cd941f4-f5b8-475c-99b8-b4bb44d8b964 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:25.206134 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd941f4-f5b8-475c-99b8-b4bb44d8b964 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:25.210368 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf358191-2e58-4a3d-a00e-aa61780fc403 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:25.211330 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf358191-2e58-4a3d-a00e-aa61780fc403 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:25.211616 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf358191-2e58-4a3d-a00e-aa61780fc403 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:25.211935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf358191-2e58-4a3d-a00e-aa61780fc403 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:25.221146 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf358191-2e58-4a3d-a00e-aa61780fc403 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:25.222538 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf358191-2e58-4a3d-a00e-aa61780fc403 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:25.232488 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1123/2232] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:25.509407 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fd1628-7f3b-4281-9911-f43a8230bca9 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:25.514174 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-11fd1628-7f3b-4281-9911-f43a8230bca9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:25.526040 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fd1628-7f3b-4281-9911-f43a8230bca9 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:25.531457 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acfbfc2-7706-4af8-99bb-4f6cf900adc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:25.532132 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acfbfc2-7706-4af8-99bb-4f6cf900adc4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:25.532408 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acfbfc2-7706-4af8-99bb-4f6cf900adc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:25.532686 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acfbfc2-7706-4af8-99bb-4f6cf900adc4 admin admin] PATH_INFO: `/v3/projects/1ca77f31e2244ba1b13d57cbcf797c20` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:25.534584 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7acfbfc2-7706-4af8-99bb-4f6cf900adc4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ca77f31e2244ba1b13d57cbcf797c20)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:25.536486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7acfbfc2-7706-4af8-99bb-4f6cf900adc4 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:25.586983 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-7acfbfc2-7706-4af8-99bb-4f6cf900adc4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ca77f31e2244ba1b13d57cbcf797c20', 'request_id': 'req-7acfbfc2-7706-4af8-99bb-4f6cf900adc4'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:25.592625 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-7acfbfc2-7706-4af8-99bb-4f6cf900adc4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ca77f31e2244ba1b13d57cbcf797c20', 'request_id': 'req-7acfbfc2-7706-4af8-99bb-4f6cf900adc4'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:25.601859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-7acfbfc2-7706-4af8-99bb-4f6cf900adc4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ca77f31e2244ba1b13d57cbcf797c20', 'request_id': 'req-7acfbfc2-7706-4af8-99bb-4f6cf900adc4'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:25.613672 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-7acfbfc2-7706-4af8-99bb-4f6cf900adc4 admin admin] The token cache is being invalidate because project 1ca77f31e2244ba1b13d57cbcf797c20 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:25.613962 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-7acfbfc2-7706-4af8-99bb-4f6cf900adc4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:25.614726 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-7acfbfc2-7706-4af8-99bb-4f6cf900adc4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:25.616501 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1110/2233] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:25 2022] DELETE /identity/v3/projects/1ca77f31e2244ba1b13d57cbcf797c20 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:25.620941 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf358191-2e58-4a3d-a00e-aa61780fc403 service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:25.631329 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf358191-2e58-4a3d-a00e-aa61780fc403 service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:25.693883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e11ee11-3d86-48a1-9b06-ff91f74126ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:25.694527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e11ee11-3d86-48a1-9b06-ff91f74126ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:25.694801 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e11ee11-3d86-48a1-9b06-ff91f74126ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:25.695076 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e11ee11-3d86-48a1-9b06-ff91f74126ee admin admin] PATH_INFO: `/v3/users/7f711122516d4ca6a3651879d7b9fff2` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:25.696342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e11ee11-3d86-48a1-9b06-ff91f74126ee admin admin] RBAC: Authorizing `identity:delete_user(user_id=7f711122516d4ca6a3651879d7b9fff2)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:25.698249 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e11ee11-3d86-48a1-9b06-ff91f74126ee admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:25.748849 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-5e11ee11-3d86-48a1-9b06-ff91f74126ee admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7f711122516d4ca6a3651879d7b9fff2'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:25.757846 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-5e11ee11-3d86-48a1-9b06-ff91f74126ee admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7f711122516d4ca6a3651879d7b9fff2'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:25.758384 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-5e11ee11-3d86-48a1-9b06-ff91f74126ee admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7f711122516d4ca6a3651879d7b9fff2'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:25.763261 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:25.772971 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-5e11ee11-3d86-48a1-9b06-ff91f74126ee admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7f711122516d4ca6a3651879d7b9fff2'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:25.780398 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-5e11ee11-3d86-48a1-9b06-ff91f74126ee admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7f711122516d4ca6a3651879d7b9fff2'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:25.786650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1124/2234] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:25 2022] DELETE /identity/v3/users/7f711122516d4ca6a3651879d7b9fff2 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:26.080603 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acfbfc2-7706-4af8-99bb-4f6cf900adc4 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:26.082055 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-7acfbfc2-7706-4af8-99bb-4f6cf900adc4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:26.096372 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acfbfc2-7706-4af8-99bb-4f6cf900adc4 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:26.156371 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de02b9a-75f8-4a26-b1a0-43ddb2a2a9f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:26.157028 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de02b9a-75f8-4a26-b1a0-43ddb2a2a9f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:26.157297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de02b9a-75f8-4a26-b1a0-43ddb2a2a9f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:26.157556 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de02b9a-75f8-4a26-b1a0-43ddb2a2a9f2 admin admin] PATH_INFO: `/v3/projects/63f1234bf985406d9886d95e81a4770e` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:26.158746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de02b9a-75f8-4a26-b1a0-43ddb2a2a9f2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=63f1234bf985406d9886d95e81a4770e)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:26.160749 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de02b9a-75f8-4a26-b1a0-43ddb2a2a9f2 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:26.200965 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:26.213779 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-3de02b9a-75f8-4a26-b1a0-43ddb2a2a9f2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '63f1234bf985406d9886d95e81a4770e', 'request_id': 'req-3de02b9a-75f8-4a26-b1a0-43ddb2a2a9f2'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:26.219499 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-3de02b9a-75f8-4a26-b1a0-43ddb2a2a9f2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '63f1234bf985406d9886d95e81a4770e', 'request_id': 'req-3de02b9a-75f8-4a26-b1a0-43ddb2a2a9f2'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:26.228502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-3de02b9a-75f8-4a26-b1a0-43ddb2a2a9f2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '63f1234bf985406d9886d95e81a4770e', 'request_id': 'req-3de02b9a-75f8-4a26-b1a0-43ddb2a2a9f2'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:26.240666 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-3de02b9a-75f8-4a26-b1a0-43ddb2a2a9f2 admin admin] The token cache is being invalidate because project 63f1234bf985406d9886d95e81a4770e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:26.240852 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-3de02b9a-75f8-4a26-b1a0-43ddb2a2a9f2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:26.241571 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-3de02b9a-75f8-4a26-b1a0-43ddb2a2a9f2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:26.243244 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1111/2235] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:26 2022] DELETE /identity/v3/projects/63f1234bf985406d9886d95e81a4770e => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:26.247525 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e11ee11-3d86-48a1-9b06-ff91f74126ee admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:26.249233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-5e11ee11-3d86-48a1-9b06-ff91f74126ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:26.262821 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e11ee11-3d86-48a1-9b06-ff91f74126ee admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:26.326424 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0405aa3a-3bb5-48be-89ef-08ba0999012f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:26.327066 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0405aa3a-3bb5-48be-89ef-08ba0999012f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:26.327340 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0405aa3a-3bb5-48be-89ef-08ba0999012f admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:26.327611 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0405aa3a-3bb5-48be-89ef-08ba0999012f admin admin] PATH_INFO: `/v3/users/aa4d8660753b4f8393acb6f684a5c1b3` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:26.328766 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0405aa3a-3bb5-48be-89ef-08ba0999012f admin admin] RBAC: Authorizing `identity:delete_user(user_id=aa4d8660753b4f8393acb6f684a5c1b3)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:26.330685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0405aa3a-3bb5-48be-89ef-08ba0999012f admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:26.380772 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-0405aa3a-3bb5-48be-89ef-08ba0999012f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aa4d8660753b4f8393acb6f684a5c1b3'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:26.390874 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-0405aa3a-3bb5-48be-89ef-08ba0999012f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aa4d8660753b4f8393acb6f684a5c1b3'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:26.391522 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-0405aa3a-3bb5-48be-89ef-08ba0999012f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aa4d8660753b4f8393acb6f684a5c1b3'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:26.403247 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-0405aa3a-3bb5-48be-89ef-08ba0999012f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aa4d8660753b4f8393acb6f684a5c1b3'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:26.409985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-0405aa3a-3bb5-48be-89ef-08ba0999012f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aa4d8660753b4f8393acb6f684a5c1b3'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:26.415972 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1125/2236] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:26 2022] DELETE /identity/v3/users/aa4d8660753b4f8393acb6f684a5c1b3 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:08:26.701132 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de02b9a-75f8-4a26-b1a0-43ddb2a2a9f2 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:26.702792 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-3de02b9a-75f8-4a26-b1a0-43ddb2a2a9f2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:26.716586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de02b9a-75f8-4a26-b1a0-43ddb2a2a9f2 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:26.775080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc877c5-a349-47f1-87a6-5c39d312d883 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:26.775752 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc877c5-a349-47f1-87a6-5c39d312d883 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:26.776019 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc877c5-a349-47f1-87a6-5c39d312d883 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:26.776291 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc877c5-a349-47f1-87a6-5c39d312d883 admin admin] PATH_INFO: `/v3/projects/18c2e00e366e496face932da216d6946` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:26.777411 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc877c5-a349-47f1-87a6-5c39d312d883 admin admin] RBAC: Authorizing `identity:delete_project(project_id=18c2e00e366e496face932da216d6946)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:26.779316 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc877c5-a349-47f1-87a6-5c39d312d883 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:26.830382 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-5dc877c5-a349-47f1-87a6-5c39d312d883 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '18c2e00e366e496face932da216d6946', 'request_id': 'req-5dc877c5-a349-47f1-87a6-5c39d312d883'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:26.836308 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-5dc877c5-a349-47f1-87a6-5c39d312d883 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '18c2e00e366e496face932da216d6946', 'request_id': 'req-5dc877c5-a349-47f1-87a6-5c39d312d883'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:26.844430 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-5dc877c5-a349-47f1-87a6-5c39d312d883 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '18c2e00e366e496face932da216d6946', 'request_id': 'req-5dc877c5-a349-47f1-87a6-5c39d312d883'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:26.855957 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-5dc877c5-a349-47f1-87a6-5c39d312d883 admin admin] The token cache is being invalidate because project 18c2e00e366e496face932da216d6946 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:26.856238 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-5dc877c5-a349-47f1-87a6-5c39d312d883 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:26.856880 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-5dc877c5-a349-47f1-87a6-5c39d312d883 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:26.858369 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1112/2237] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:26 2022] DELETE /identity/v3/projects/18c2e00e366e496face932da216d6946 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:26.863737 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3620b38b-9e94-4714-aef9-1ca227959193 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:26.864057 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3620b38b-9e94-4714-aef9-1ca227959193 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:26.864295 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3620b38b-9e94-4714-aef9-1ca227959193 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:26.884829 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-3620b38b-9e94-4714-aef9-1ca227959193 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:26.886029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3620b38b-9e94-4714-aef9-1ca227959193 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:26.942748 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:26.956204 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1126/2238] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:26.964582 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f71b0f3-24bd-4bdb-80f3-591579d8695c None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:26.964863 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f71b0f3-24bd-4bdb-80f3-591579d8695c None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:26.965133 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f71b0f3-24bd-4bdb-80f3-591579d8695c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:26.985733 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-9f71b0f3-24bd-4bdb-80f3-591579d8695c None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:26.986941 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-9f71b0f3-24bd-4bdb-80f3-591579d8695c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:26.999492 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1113/2239] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:27.000270 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3620b38b-9e94-4714-aef9-1ca227959193 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:27.010464 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3620b38b-9e94-4714-aef9-1ca227959193 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:27.014949 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09af67b-d37a-415d-8eb9-68f579b88086 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:27.015650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09af67b-d37a-415d-8eb9-68f579b88086 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:27.016038 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09af67b-d37a-415d-8eb9-68f579b88086 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:27.016361 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09af67b-d37a-415d-8eb9-68f579b88086 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:27.017074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09af67b-d37a-415d-8eb9-68f579b88086 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:08:27.017375 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09af67b-d37a-415d-8eb9-68f579b88086 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:27.018948 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09af67b-d37a-415d-8eb9-68f579b88086 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:27.034707 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1127/2240] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:08:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 01:08:27.035700 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f71b0f3-24bd-4bdb-80f3-591579d8695c None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:27.045582 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f71b0f3-24bd-4bdb-80f3-591579d8695c None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:27.050544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12221c2a-9d44-48e4-ac7e-bb81f3ff3382 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:27.050865 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12221c2a-9d44-48e4-ac7e-bb81f3ff3382 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:27.051140 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12221c2a-9d44-48e4-ac7e-bb81f3ff3382 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:27.051404 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12221c2a-9d44-48e4-ac7e-bb81f3ff3382 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:27.052079 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12221c2a-9d44-48e4-ac7e-bb81f3ff3382 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:27.053980 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12221c2a-9d44-48e4-ac7e-bb81f3ff3382 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:27.089646 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1114/2241] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:27 2022] POST /identity/v3/projects => generated 383 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:27.090355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09af67b-d37a-415d-8eb9-68f579b88086 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:27.100475 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09af67b-d37a-415d-8eb9-68f579b88086 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:27.157082 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b4d484-8fa5-4d7f-b33e-8d5d94043bb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:27.158033 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b4d484-8fa5-4d7f-b33e-8d5d94043bb2 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:27.158033 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b4d484-8fa5-4d7f-b33e-8d5d94043bb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:27.158251 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b4d484-8fa5-4d7f-b33e-8d5d94043bb2 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:27.158815 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b4d484-8fa5-4d7f-b33e-8d5d94043bb2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:27.160781 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b4d484-8fa5-4d7f-b33e-8d5d94043bb2 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:27.183301 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1128/2242] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:27 2022] POST /identity/v3/users => generated 292 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:08:27.187542 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12221c2a-9d44-48e4-ac7e-bb81f3ff3382 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:27.198171 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12221c2a-9d44-48e4-ac7e-bb81f3ff3382 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:27.203737 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f604caf-3219-44ea-9661-2d7c0f418b72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:27.204407 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f604caf-3219-44ea-9661-2d7c0f418b72 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:27.204748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f604caf-3219-44ea-9661-2d7c0f418b72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:27.205145 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f604caf-3219-44ea-9661-2d7c0f418b72 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:27.205973 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f604caf-3219-44ea-9661-2d7c0f418b72 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:27.207876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f604caf-3219-44ea-9661-2d7c0f418b72 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:27.218854 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1115/2243] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:27 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:27.222899 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b4d484-8fa5-4d7f-b33e-8d5d94043bb2 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:27.233033 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b4d484-8fa5-4d7f-b33e-8d5d94043bb2 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:27.237514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc7b261-22ba-4642-b387-11b99f7e50c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:27.238110 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc7b261-22ba-4642-b387-11b99f7e50c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:27.238378 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc7b261-22ba-4642-b387-11b99f7e50c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:27.238638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc7b261-22ba-4642-b387-11b99f7e50c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:27.239268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc7b261-22ba-4642-b387-11b99f7e50c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:27.240861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc7b261-22ba-4642-b387-11b99f7e50c7 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:27.251619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1129/2244] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:27 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:27.255889 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f604caf-3219-44ea-9661-2d7c0f418b72 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:27.266146 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f604caf-3219-44ea-9661-2d7c0f418b72 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:27.271057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefb1d41-05b8-4699-8db1-ac9b50c1e6c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:27.271728 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefb1d41-05b8-4699-8db1-ac9b50c1e6c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:27.272065 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefb1d41-05b8-4699-8db1-ac9b50c1e6c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:27.272336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefb1d41-05b8-4699-8db1-ac9b50c1e6c4 admin admin] PATH_INFO: `/v3/projects/995977e0c3de415a9d1415d0de064c95/users/04de1fb89db441f7b52c2be66d726a5c/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:27.284379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:DELETE=12,keystone:SELECT=85,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:27.288338 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefb1d41-05b8-4699-8db1-ac9b50c1e6c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=995977e0c3de415a9d1415d0de064c95, user_id=04de1fb89db441f7b52c2be66d726a5c, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:27.291510 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefb1d41-05b8-4699-8db1-ac9b50c1e6c4 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:27.308213 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1116/2245] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:27 2022] PUT /identity/v3/projects/995977e0c3de415a9d1415d0de064c95/users/04de1fb89db441f7b52c2be66d726a5c/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:27.318848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274602bc-a169-4a08-8d85-6795ffb21bf2 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:27.319157 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274602bc-a169-4a08-8d85-6795ffb21bf2 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:27.319502 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274602bc-a169-4a08-8d85-6795ffb21bf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:27.349844 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-274602bc-a169-4a08-8d85-6795ffb21bf2 None None] MFA Rules not processed for user `04de1fb89db441f7b52c2be66d726a5c`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:27.351124 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-274602bc-a169-4a08-8d85-6795ffb21bf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:27.419498 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1130/2246] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:27 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:27.431592 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefb1d41-05b8-4699-8db1-ac9b50c1e6c4 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:27.433650 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-eefb1d41-05b8-4699-8db1-ac9b50c1e6c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:27.503232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefb1d41-05b8-4699-8db1-ac9b50c1e6c4 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:27.507402 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a73ac9b-ec53-44a5-ae4f-d1c388733740 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:27.508339 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a73ac9b-ec53-44a5-ae4f-d1c388733740 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:27.508672 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a73ac9b-ec53-44a5-ae4f-d1c388733740 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:27.508944 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a73ac9b-ec53-44a5-ae4f-d1c388733740 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:27.519076 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a73ac9b-ec53-44a5-ae4f-d1c388733740 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:27.520641 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a73ac9b-ec53-44a5-ae4f-d1c388733740 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:27.530648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1117/2247] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:27 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:30.171333 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274602bc-a169-4a08-8d85-6795ffb21bf2 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:30.231895 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:30.244591 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274602bc-a169-4a08-8d85-6795ffb21bf2 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:30.249493 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56501e09-a4bb-428e-9c1b-352eb0b3abe3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:30.250101 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56501e09-a4bb-428e-9c1b-352eb0b3abe3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:30.250370 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56501e09-a4bb-428e-9c1b-352eb0b3abe3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:30.250636 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56501e09-a4bb-428e-9c1b-352eb0b3abe3 admin admin] PATH_INFO: `/v3/users/04de1fb89db441f7b52c2be66d726a5c` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:30.251786 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56501e09-a4bb-428e-9c1b-352eb0b3abe3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=04de1fb89db441f7b52c2be66d726a5c)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:30.253702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56501e09-a4bb-428e-9c1b-352eb0b3abe3 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:30.309809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-56501e09-a4bb-428e-9c1b-352eb0b3abe3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '04de1fb89db441f7b52c2be66d726a5c'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:30.318857 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-56501e09-a4bb-428e-9c1b-352eb0b3abe3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '04de1fb89db441f7b52c2be66d726a5c'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:30.319479 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-56501e09-a4bb-428e-9c1b-352eb0b3abe3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '04de1fb89db441f7b52c2be66d726a5c'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:30.332288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-56501e09-a4bb-428e-9c1b-352eb0b3abe3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '04de1fb89db441f7b52c2be66d726a5c'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:30.342436 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-56501e09-a4bb-428e-9c1b-352eb0b3abe3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '04de1fb89db441f7b52c2be66d726a5c'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:30.350107 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1131/2248] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:30 2022] DELETE /identity/v3/users/04de1fb89db441f7b52c2be66d726a5c => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:30.355023 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10b76be-a1a3-4331-8800-0ab376c91cde None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:30.355535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10b76be-a1a3-4331-8800-0ab376c91cde None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:30.355660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10b76be-a1a3-4331-8800-0ab376c91cde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:30.375343 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-d10b76be-a1a3-4331-8800-0ab376c91cde None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:30.376627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d10b76be-a1a3-4331-8800-0ab376c91cde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:30.440361 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1118/2249] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:30.450949 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56501e09-a4bb-428e-9c1b-352eb0b3abe3 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:30.452531 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-56501e09-a4bb-428e-9c1b-352eb0b3abe3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:30.519012 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56501e09-a4bb-428e-9c1b-352eb0b3abe3 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:30.524155 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a488ff-7e01-4225-9d48-3fa725f2f66e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:30.524803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a488ff-7e01-4225-9d48-3fa725f2f66e service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:30.525080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a488ff-7e01-4225-9d48-3fa725f2f66e service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:30.525344 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a488ff-7e01-4225-9d48-3fa725f2f66e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:30.535580 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a488ff-7e01-4225-9d48-3fa725f2f66e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:30.537048 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a488ff-7e01-4225-9d48-3fa725f2f66e service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:30.546815 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1132/2250] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:30.820662 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10b76be-a1a3-4331-8800-0ab376c91cde None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:30.822196 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d10b76be-a1a3-4331-8800-0ab376c91cde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:30.837355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10b76be-a1a3-4331-8800-0ab376c91cde None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:30.842250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de601b0-4fc4-4710-83fe-e274aa8483d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:30.842965 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de601b0-4fc4-4710-83fe-e274aa8483d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:30.843265 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de601b0-4fc4-4710-83fe-e274aa8483d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:30.843533 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de601b0-4fc4-4710-83fe-e274aa8483d8 admin admin] PATH_INFO: `/v3/projects/995977e0c3de415a9d1415d0de064c95` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:30.844734 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de601b0-4fc4-4710-83fe-e274aa8483d8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=995977e0c3de415a9d1415d0de064c95)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:30.846674 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de601b0-4fc4-4710-83fe-e274aa8483d8 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:30.897539 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-8de601b0-4fc4-4710-83fe-e274aa8483d8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '995977e0c3de415a9d1415d0de064c95', 'request_id': 'req-8de601b0-4fc4-4710-83fe-e274aa8483d8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:30.903188 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-8de601b0-4fc4-4710-83fe-e274aa8483d8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '995977e0c3de415a9d1415d0de064c95', 'request_id': 'req-8de601b0-4fc4-4710-83fe-e274aa8483d8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:30.913633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-8de601b0-4fc4-4710-83fe-e274aa8483d8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '995977e0c3de415a9d1415d0de064c95', 'request_id': 'req-8de601b0-4fc4-4710-83fe-e274aa8483d8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:30.925061 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-8de601b0-4fc4-4710-83fe-e274aa8483d8 admin admin] The token cache is being invalidate because project 995977e0c3de415a9d1415d0de064c95 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:30.925351 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-8de601b0-4fc4-4710-83fe-e274aa8483d8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:30.926021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-8de601b0-4fc4-4710-83fe-e274aa8483d8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:30.927571 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1119/2251] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:30 2022] DELETE /identity/v3/projects/995977e0c3de415a9d1415d0de064c95 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:30.932771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526de5f6-9a67-4bd4-a2d1-26ff91cbb999 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:30.933130 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526de5f6-9a67-4bd4-a2d1-26ff91cbb999 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:30.933406 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526de5f6-9a67-4bd4-a2d1-26ff91cbb999 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:30.955878 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-526de5f6-9a67-4bd4-a2d1-26ff91cbb999 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:30.957059 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-526de5f6-9a67-4bd4-a2d1-26ff91cbb999 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:31.021368 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1133/2252] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:08:31.031238 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb61bb39-571e-4d3c-b76a-07132187fbf3 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:31.031519 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb61bb39-571e-4d3c-b76a-07132187fbf3 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:31.031820 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb61bb39-571e-4d3c-b76a-07132187fbf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:31.053209 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-fb61bb39-571e-4d3c-b76a-07132187fbf3 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:31.054452 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-fb61bb39-571e-4d3c-b76a-07132187fbf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:31.063931 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1120/2253] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:31.068979 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526de5f6-9a67-4bd4-a2d1-26ff91cbb999 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:31.082700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526de5f6-9a67-4bd4-a2d1-26ff91cbb999 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:31.090975 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42b72fd-429a-47bd-b278-32d47b732251 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:31.091587 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42b72fd-429a-47bd-b278-32d47b732251 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:31.091874 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42b72fd-429a-47bd-b278-32d47b732251 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:31.092151 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42b72fd-429a-47bd-b278-32d47b732251 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:31.092786 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42b72fd-429a-47bd-b278-32d47b732251 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:08:31.093065 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42b72fd-429a-47bd-b278-32d47b732251 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:31.094641 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42b72fd-429a-47bd-b278-32d47b732251 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:31.103570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:31.108268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1134/2254] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:08:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 01:08:31.112639 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb61bb39-571e-4d3c-b76a-07132187fbf3 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:31.123192 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb61bb39-571e-4d3c-b76a-07132187fbf3 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:31.127544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db84af2b-f31f-4339-a43b-0044c7923c18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:31.128187 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db84af2b-f31f-4339-a43b-0044c7923c18 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:31.128453 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db84af2b-f31f-4339-a43b-0044c7923c18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:31.128717 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db84af2b-f31f-4339-a43b-0044c7923c18 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:31.129365 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db84af2b-f31f-4339-a43b-0044c7923c18 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:31.131300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db84af2b-f31f-4339-a43b-0044c7923c18 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:31.136956 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:31.167696 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1121/2255] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:31 2022] POST /identity/v3/projects => generated 399 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:31.169591 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42b72fd-429a-47bd-b278-32d47b732251 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:31.178941 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42b72fd-429a-47bd-b278-32d47b732251 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:31.241989 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7bb2e3-c315-402e-80f6-08cde3b7ad96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:31.242697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7bb2e3-c315-402e-80f6-08cde3b7ad96 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:31.243039 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7bb2e3-c315-402e-80f6-08cde3b7ad96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:31.243372 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7bb2e3-c315-402e-80f6-08cde3b7ad96 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:31.244108 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7bb2e3-c315-402e-80f6-08cde3b7ad96 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:31.246414 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7bb2e3-c315-402e-80f6-08cde3b7ad96 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:31.269645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1135/2256] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:31 2022] POST /identity/v3/users => generated 300 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:08:31.274103 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db84af2b-f31f-4339-a43b-0044c7923c18 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:31.284541 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db84af2b-f31f-4339-a43b-0044c7923c18 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:31.289120 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de98b71-964f-4657-a352-b4e4570c83f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:31.289713 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de98b71-964f-4657-a352-b4e4570c83f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:31.289985 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de98b71-964f-4657-a352-b4e4570c83f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:31.290255 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de98b71-964f-4657-a352-b4e4570c83f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:31.290922 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de98b71-964f-4657-a352-b4e4570c83f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:31.292610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de98b71-964f-4657-a352-b4e4570c83f7 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:31.302533 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1122/2257] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:31 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:31.306736 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7bb2e3-c315-402e-80f6-08cde3b7ad96 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:31.317173 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7bb2e3-c315-402e-80f6-08cde3b7ad96 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:31.322093 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f593bb0e-c273-442f-bc52-f55d1cdc46fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:31.322694 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f593bb0e-c273-442f-bc52-f55d1cdc46fd admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:31.322967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f593bb0e-c273-442f-bc52-f55d1cdc46fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:31.323268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f593bb0e-c273-442f-bc52-f55d1cdc46fd admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:31.323998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f593bb0e-c273-442f-bc52-f55d1cdc46fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:31.325592 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f593bb0e-c273-442f-bc52-f55d1cdc46fd admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:31.336762 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1136/2258] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:31 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:31.341267 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de98b71-964f-4657-a352-b4e4570c83f7 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:31.352024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de98b71-964f-4657-a352-b4e4570c83f7 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:31.357547 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74960846-7440-401b-8977-b9191a775cfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:31.358199 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74960846-7440-401b-8977-b9191a775cfb admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:31.358467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74960846-7440-401b-8977-b9191a775cfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:31.358731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74960846-7440-401b-8977-b9191a775cfb admin admin] PATH_INFO: `/v3/projects/e419a44252d8478cb002db0af2e27bc4/users/d72f96923f3d45a099d2718a6af38cc4/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:31.374297 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74960846-7440-401b-8977-b9191a775cfb admin admin] RBAC: Authorizing `identity:create_grant(project_id=e419a44252d8478cb002db0af2e27bc4, user_id=d72f96923f3d45a099d2718a6af38cc4, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:31.376972 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74960846-7440-401b-8977-b9191a775cfb admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:31.392605 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1123/2259] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:31 2022] PUT /identity/v3/projects/e419a44252d8478cb002db0af2e27bc4/users/d72f96923f3d45a099d2718a6af38cc4/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:31.403083 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f52fa57-1923-435a-823f-3ac31cc858fe None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:31.403360 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f52fa57-1923-435a-823f-3ac31cc858fe None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:31.403625 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f52fa57-1923-435a-823f-3ac31cc858fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:31.432547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-5f52fa57-1923-435a-823f-3ac31cc858fe None None] MFA Rules not processed for user `d72f96923f3d45a099d2718a6af38cc4`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:31.433892 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-5f52fa57-1923-435a-823f-3ac31cc858fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:31.496937 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1137/2260] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:31 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:31.508699 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74960846-7440-401b-8977-b9191a775cfb admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:31.510768 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-74960846-7440-401b-8977-b9191a775cfb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:31.570317 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74960846-7440-401b-8977-b9191a775cfb admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:31.574786 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a536d74-b849-4d4d-b92e-f799b2dda910 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:31.575403 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a536d74-b849-4d4d-b92e-f799b2dda910 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:31.575677 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a536d74-b849-4d4d-b92e-f799b2dda910 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:31.575977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a536d74-b849-4d4d-b92e-f799b2dda910 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:31.586232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a536d74-b849-4d4d-b92e-f799b2dda910 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:31.587668 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a536d74-b849-4d4d-b92e-f799b2dda910 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:31.597318 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1124/2261] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:31 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:33.675374 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f52fa57-1923-435a-823f-3ac31cc858fe None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:33.676993 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-5f52fa57-1923-435a-823f-3ac31cc858fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:33.740806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f52fa57-1923-435a-823f-3ac31cc858fe None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:33.745207 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55db0c51-d5e5-4d22-be61-5db7b477752e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:33.746038 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55db0c51-d5e5-4d22-be61-5db7b477752e service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:33.746346 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55db0c51-d5e5-4d22-be61-5db7b477752e service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:33.746573 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55db0c51-d5e5-4d22-be61-5db7b477752e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:33.748979 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-55db0c51-d5e5-4d22-be61-5db7b477752e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:33.777054 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:33.823148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55db0c51-d5e5-4d22-be61-5db7b477752e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:33.824637 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55db0c51-d5e5-4d22-be61-5db7b477752e service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:33.835047 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1138/2262] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:36.990556 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a536d74-b849-4d4d-b92e-f799b2dda910 service cinder] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:36.992249 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-7a536d74-b849-4d4d-b92e-f799b2dda910 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:37.007129 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a536d74-b849-4d4d-b92e-f799b2dda910 service cinder] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:37.011861 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739ab5d9-af9c-450f-b9ef-52a8d577dbee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:37.012539 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739ab5d9-af9c-450f-b9ef-52a8d577dbee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:37.012940 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739ab5d9-af9c-450f-b9ef-52a8d577dbee admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:37.013214 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739ab5d9-af9c-450f-b9ef-52a8d577dbee admin admin] PATH_INFO: `/v3/users/ecaf8ec105c04b919256f852da859eba` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:37.014331 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739ab5d9-af9c-450f-b9ef-52a8d577dbee admin admin] RBAC: Authorizing `identity:delete_user(user_id=ecaf8ec105c04b919256f852da859eba)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:37.016274 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739ab5d9-af9c-450f-b9ef-52a8d577dbee admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:37.067264 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-739ab5d9-af9c-450f-b9ef-52a8d577dbee admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ecaf8ec105c04b919256f852da859eba'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:37.076176 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-739ab5d9-af9c-450f-b9ef-52a8d577dbee admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ecaf8ec105c04b919256f852da859eba'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:37.076958 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-739ab5d9-af9c-450f-b9ef-52a8d577dbee admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ecaf8ec105c04b919256f852da859eba'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:37.084527 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-739ab5d9-af9c-450f-b9ef-52a8d577dbee admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ecaf8ec105c04b919256f852da859eba'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:37.097468 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-739ab5d9-af9c-450f-b9ef-52a8d577dbee admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ecaf8ec105c04b919256f852da859eba'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:37.103329 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1125/2263] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:36 2022] DELETE /identity/v3/users/ecaf8ec105c04b919256f852da859eba => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:37.288797 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55db0c51-d5e5-4d22-be61-5db7b477752e service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:37.299435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55db0c51-d5e5-4d22-be61-5db7b477752e service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:37.359883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a745997f-1e98-4605-be3e-26ed7e8e3451 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:37.360514 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a745997f-1e98-4605-be3e-26ed7e8e3451 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:37.360783 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a745997f-1e98-4605-be3e-26ed7e8e3451 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:37.361053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a745997f-1e98-4605-be3e-26ed7e8e3451 admin admin] PATH_INFO: `/v3/projects/6b68b886c19c45d585409651f33b8626` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:37.362268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a745997f-1e98-4605-be3e-26ed7e8e3451 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6b68b886c19c45d585409651f33b8626)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:37.364233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a745997f-1e98-4605-be3e-26ed7e8e3451 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:37.415356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a745997f-1e98-4605-be3e-26ed7e8e3451 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6b68b886c19c45d585409651f33b8626', 'request_id': 'req-a745997f-1e98-4605-be3e-26ed7e8e3451'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:37.425162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a745997f-1e98-4605-be3e-26ed7e8e3451 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6b68b886c19c45d585409651f33b8626', 'request_id': 'req-a745997f-1e98-4605-be3e-26ed7e8e3451'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:37.430733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a745997f-1e98-4605-be3e-26ed7e8e3451 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6b68b886c19c45d585409651f33b8626', 'request_id': 'req-a745997f-1e98-4605-be3e-26ed7e8e3451'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:37.442059 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a745997f-1e98-4605-be3e-26ed7e8e3451 admin admin] The token cache is being invalidate because project 6b68b886c19c45d585409651f33b8626 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:37.442339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a745997f-1e98-4605-be3e-26ed7e8e3451 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:37.442972 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a745997f-1e98-4605-be3e-26ed7e8e3451 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:37.444497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1139/2264] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:37 2022] DELETE /identity/v3/projects/6b68b886c19c45d585409651f33b8626 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:37.449712 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffd412e-39c3-4517-9ee6-df8edbd8ab48 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:37.450124 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffd412e-39c3-4517-9ee6-df8edbd8ab48 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:37.450483 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffd412e-39c3-4517-9ee6-df8edbd8ab48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:37.471595 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-dffd412e-39c3-4517-9ee6-df8edbd8ab48 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:37.472837 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-dffd412e-39c3-4517-9ee6-df8edbd8ab48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:37.494378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:37.540173 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1126/2265] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:37.548359 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93c3a51-82c8-469f-ad7e-f1b5d9c6a467 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:37.548623 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93c3a51-82c8-469f-ad7e-f1b5d9c6a467 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:37.548887 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93c3a51-82c8-469f-ad7e-f1b5d9c6a467 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:37.568560 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-e93c3a51-82c8-469f-ad7e-f1b5d9c6a467 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:37.569765 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e93c3a51-82c8-469f-ad7e-f1b5d9c6a467 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:37.577956 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1140/2266] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:08:37.582631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffd412e-39c3-4517-9ee6-df8edbd8ab48 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:37.592747 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffd412e-39c3-4517-9ee6-df8edbd8ab48 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:37.597664 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3eb5e5-df7c-462d-bf60-f79d2b73792c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:37.598267 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3eb5e5-df7c-462d-bf60-f79d2b73792c admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:37.598533 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3eb5e5-df7c-462d-bf60-f79d2b73792c admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:37.598795 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3eb5e5-df7c-462d-bf60-f79d2b73792c admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:37.599471 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3eb5e5-df7c-462d-bf60-f79d2b73792c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:08:37.599822 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3eb5e5-df7c-462d-bf60-f79d2b73792c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:37.601378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3eb5e5-df7c-462d-bf60-f79d2b73792c admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:37.613713 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1127/2267] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:08:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 01:08:37.618244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93c3a51-82c8-469f-ad7e-f1b5d9c6a467 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:37.627605 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93c3a51-82c8-469f-ad7e-f1b5d9c6a467 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:37.632331 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8993f015-95fa-4dc8-ba70-379f16f55758 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:37.632925 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8993f015-95fa-4dc8-ba70-379f16f55758 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:37.633194 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8993f015-95fa-4dc8-ba70-379f16f55758 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:37.633454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8993f015-95fa-4dc8-ba70-379f16f55758 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:37.634068 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8993f015-95fa-4dc8-ba70-379f16f55758 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:37.635987 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8993f015-95fa-4dc8-ba70-379f16f55758 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:37.665116 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1141/2268] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:37 2022] POST /identity/v3/projects => generated 375 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:37.669545 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3eb5e5-df7c-462d-bf60-f79d2b73792c admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:37.679896 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3eb5e5-df7c-462d-bf60-f79d2b73792c admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:37.737296 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38172558-4f38-48e1-8483-eb61227749c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:37.737925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38172558-4f38-48e1-8483-eb61227749c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:37.738192 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38172558-4f38-48e1-8483-eb61227749c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:37.738460 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38172558-4f38-48e1-8483-eb61227749c7 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:37.739068 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38172558-4f38-48e1-8483-eb61227749c7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:37.741076 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38172558-4f38-48e1-8483-eb61227749c7 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:37.764200 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1128/2269] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:37 2022] POST /identity/v3/users => generated 288 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:37.768532 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8993f015-95fa-4dc8-ba70-379f16f55758 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:37.779255 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8993f015-95fa-4dc8-ba70-379f16f55758 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:37.784728 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82f188f-be1e-401f-bc02-9972403f2d2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:37.785329 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82f188f-be1e-401f-bc02-9972403f2d2f admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:37.785600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82f188f-be1e-401f-bc02-9972403f2d2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:37.785863 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82f188f-be1e-401f-bc02-9972403f2d2f admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:37.786491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82f188f-be1e-401f-bc02-9972403f2d2f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:37.788147 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82f188f-be1e-401f-bc02-9972403f2d2f admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:37.798600 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1142/2270] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:37 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:37.802711 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38172558-4f38-48e1-8483-eb61227749c7 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:37.813289 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38172558-4f38-48e1-8483-eb61227749c7 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:37.818491 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a9f937-e74c-4402-b8c6-a63b4a7de2b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:37.819123 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a9f937-e74c-4402-b8c6-a63b4a7de2b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:37.819393 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a9f937-e74c-4402-b8c6-a63b4a7de2b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:37.819658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a9f937-e74c-4402-b8c6-a63b4a7de2b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:37.820359 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a9f937-e74c-4402-b8c6-a63b4a7de2b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:37.821982 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a9f937-e74c-4402-b8c6-a63b4a7de2b9 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:37.834039 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1129/2271] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:37 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:37.838845 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82f188f-be1e-401f-bc02-9972403f2d2f admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:37.850022 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82f188f-be1e-401f-bc02-9972403f2d2f admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:37.854731 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570909d2-bec0-4283-85d1-7abd1b6f12fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:37.855377 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570909d2-bec0-4283-85d1-7abd1b6f12fc admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:37.855655 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570909d2-bec0-4283-85d1-7abd1b6f12fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:37.855955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570909d2-bec0-4283-85d1-7abd1b6f12fc admin admin] PATH_INFO: `/v3/projects/4cb13eb3ad574ce9a9150be4773c00fa/users/25c6ea71c72747c59d16087ace588111/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:37.859015 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:37.874690 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570909d2-bec0-4283-85d1-7abd1b6f12fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=4cb13eb3ad574ce9a9150be4773c00fa, user_id=25c6ea71c72747c59d16087ace588111, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:37.877792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570909d2-bec0-4283-85d1-7abd1b6f12fc admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:37.891791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1143/2272] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:37 2022] PUT /identity/v3/projects/4cb13eb3ad574ce9a9150be4773c00fa/users/25c6ea71c72747c59d16087ace588111/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:37.903446 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14af53a2-fda7-445d-bf65-6b3dd5c07fc3 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:37.903741 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14af53a2-fda7-445d-bf65-6b3dd5c07fc3 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:37.904039 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14af53a2-fda7-445d-bf65-6b3dd5c07fc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:37.935333 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-14af53a2-fda7-445d-bf65-6b3dd5c07fc3 None None] MFA Rules not processed for user `25c6ea71c72747c59d16087ace588111`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:37.936583 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-14af53a2-fda7-445d-bf65-6b3dd5c07fc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:38.004565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1130/2273] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:37 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:08:38.016422 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570909d2-bec0-4283-85d1-7abd1b6f12fc admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:38.018790 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-570909d2-bec0-4283-85d1-7abd1b6f12fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:38.096694 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570909d2-bec0-4283-85d1-7abd1b6f12fc admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:38.102834 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6789bb4-ae0d-421b-a707-5a7aab7420b7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:38.103444 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6789bb4-ae0d-421b-a707-5a7aab7420b7 service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:38.103708 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6789bb4-ae0d-421b-a707-5a7aab7420b7 service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:38.104031 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6789bb4-ae0d-421b-a707-5a7aab7420b7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:38.112637 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6789bb4-ae0d-421b-a707-5a7aab7420b7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:38.114262 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6789bb4-ae0d-421b-a707-5a7aab7420b7 service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:38.126124 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1144/2274] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:38 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:38.361839 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14af53a2-fda7-445d-bf65-6b3dd5c07fc3 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:38.390919 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:38.429601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14af53a2-fda7-445d-bf65-6b3dd5c07fc3 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:38.434167 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d7ca56-21ba-4b8f-b731-6b4ade7bf720 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:38.434902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d7ca56-21ba-4b8f-b731-6b4ade7bf720 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:38.435368 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d7ca56-21ba-4b8f-b731-6b4ade7bf720 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:38.435871 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d7ca56-21ba-4b8f-b731-6b4ade7bf720 admin admin] PATH_INFO: `/v3/users/25c6ea71c72747c59d16087ace588111` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:38.437013 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d7ca56-21ba-4b8f-b731-6b4ade7bf720 admin admin] RBAC: Authorizing `identity:delete_user(user_id=25c6ea71c72747c59d16087ace588111)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:38.438935 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d7ca56-21ba-4b8f-b731-6b4ade7bf720 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:38.487213 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-90d7ca56-21ba-4b8f-b731-6b4ade7bf720 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '25c6ea71c72747c59d16087ace588111'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:38.496187 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-90d7ca56-21ba-4b8f-b731-6b4ade7bf720 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '25c6ea71c72747c59d16087ace588111'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:38.496864 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-90d7ca56-21ba-4b8f-b731-6b4ade7bf720 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '25c6ea71c72747c59d16087ace588111'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:38.504547 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-90d7ca56-21ba-4b8f-b731-6b4ade7bf720 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '25c6ea71c72747c59d16087ace588111'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:38.519799 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-90d7ca56-21ba-4b8f-b731-6b4ade7bf720 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '25c6ea71c72747c59d16087ace588111'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:38.526381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1131/2275] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:38 2022] DELETE /identity/v3/users/25c6ea71c72747c59d16087ace588111 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:38.531298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e7e7e4-142f-443a-ade9-058238074305 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:38.531823 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e7e7e4-142f-443a-ade9-058238074305 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:38.531823 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e7e7e4-142f-443a-ade9-058238074305 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:38.551962 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-e9e7e7e4-142f-443a-ade9-058238074305 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:38.553171 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e9e7e7e4-142f-443a-ade9-058238074305 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:38.616755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1145/2276] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:38.627133 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d7ca56-21ba-4b8f-b731-6b4ade7bf720 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:38.628567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-90d7ca56-21ba-4b8f-b731-6b4ade7bf720 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:38.692315 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d7ca56-21ba-4b8f-b731-6b4ade7bf720 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:38.696431 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95dde40b-f5e1-4d0a-a082-46522de5e24f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:38.697037 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95dde40b-f5e1-4d0a-a082-46522de5e24f service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:38.697305 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95dde40b-f5e1-4d0a-a082-46522de5e24f service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:38.697569 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95dde40b-f5e1-4d0a-a082-46522de5e24f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:38.707287 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95dde40b-f5e1-4d0a-a082-46522de5e24f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:38.708862 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95dde40b-f5e1-4d0a-a082-46522de5e24f service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:38.719904 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1132/2277] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:39.288540 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e7e7e4-142f-443a-ade9-058238074305 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:39.290083 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e9e7e7e4-142f-443a-ade9-058238074305 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:39.303898 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e7e7e4-142f-443a-ade9-058238074305 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:39.309758 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7965bc58-4128-4ddc-8e63-e2fdd85d0381 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:39.311248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7965bc58-4128-4ddc-8e63-e2fdd85d0381 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:39.311557 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7965bc58-4128-4ddc-8e63-e2fdd85d0381 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:39.311855 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7965bc58-4128-4ddc-8e63-e2fdd85d0381 admin admin] PATH_INFO: `/v3/projects/4cb13eb3ad574ce9a9150be4773c00fa` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:39.313167 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7965bc58-4128-4ddc-8e63-e2fdd85d0381 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4cb13eb3ad574ce9a9150be4773c00fa)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:39.315132 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7965bc58-4128-4ddc-8e63-e2fdd85d0381 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:39.384901 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7965bc58-4128-4ddc-8e63-e2fdd85d0381 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4cb13eb3ad574ce9a9150be4773c00fa', 'request_id': 'req-7965bc58-4128-4ddc-8e63-e2fdd85d0381'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:39.394184 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7965bc58-4128-4ddc-8e63-e2fdd85d0381 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4cb13eb3ad574ce9a9150be4773c00fa', 'request_id': 'req-7965bc58-4128-4ddc-8e63-e2fdd85d0381'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:39.400476 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7965bc58-4128-4ddc-8e63-e2fdd85d0381 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4cb13eb3ad574ce9a9150be4773c00fa', 'request_id': 'req-7965bc58-4128-4ddc-8e63-e2fdd85d0381'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:39.401692 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:39.414737 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7965bc58-4128-4ddc-8e63-e2fdd85d0381 admin admin] The token cache is being invalidate because project 4cb13eb3ad574ce9a9150be4773c00fa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:39.415256 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7965bc58-4128-4ddc-8e63-e2fdd85d0381 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:39.415949 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-7965bc58-4128-4ddc-8e63-e2fdd85d0381 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:39.417520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1146/2278] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:39 2022] DELETE /identity/v3/projects/4cb13eb3ad574ce9a9150be4773c00fa => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:39.423137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2a9658-49af-4cd6-b199-139d58a0d97b None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:39.423642 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2a9658-49af-4cd6-b199-139d58a0d97b None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:39.423944 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2a9658-49af-4cd6-b199-139d58a0d97b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:39.446152 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-fe2a9658-49af-4cd6-b199-139d58a0d97b None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:39.447412 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-fe2a9658-49af-4cd6-b199-139d58a0d97b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:39.489942 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:39.513462 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1133/2279] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:39.521721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed251618-538f-476f-89e8-faba1392062c None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:39.521956 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed251618-538f-476f-89e8-faba1392062c None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:39.522348 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed251618-538f-476f-89e8-faba1392062c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:39.549622 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-ed251618-538f-476f-89e8-faba1392062c None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:39.552371 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ed251618-538f-476f-89e8-faba1392062c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:39.560728 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1147/2280] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:39.565348 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2a9658-49af-4cd6-b199-139d58a0d97b None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:39.575731 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2a9658-49af-4cd6-b199-139d58a0d97b None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:39.580692 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c22301-217d-426b-92ac-b14c610d9e67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:39.581296 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c22301-217d-426b-92ac-b14c610d9e67 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:39.581541 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c22301-217d-426b-92ac-b14c610d9e67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:39.581828 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c22301-217d-426b-92ac-b14c610d9e67 admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:39.582488 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c22301-217d-426b-92ac-b14c610d9e67 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:08:39.582780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c22301-217d-426b-92ac-b14c610d9e67 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:39.584506 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c22301-217d-426b-92ac-b14c610d9e67 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:39.601163 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1134/2281] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:08:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 01:08:39.602599 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed251618-538f-476f-89e8-faba1392062c None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:39.613227 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed251618-538f-476f-89e8-faba1392062c None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:39.618164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5e192f-6919-4df1-9b8e-0deb798b250b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:39.618846 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5e192f-6919-4df1-9b8e-0deb798b250b admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:39.619187 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5e192f-6919-4df1-9b8e-0deb798b250b admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:39.619524 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5e192f-6919-4df1-9b8e-0deb798b250b admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:39.620261 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a5e192f-6919-4df1-9b8e-0deb798b250b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:39.622316 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a5e192f-6919-4df1-9b8e-0deb798b250b admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:39.654549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1148/2282] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:39 2022] POST /identity/v3/projects => generated 373 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:08:39.659152 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c22301-217d-426b-92ac-b14c610d9e67 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:39.671645 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c22301-217d-426b-92ac-b14c610d9e67 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:39.753903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c994c3-b987-472a-99f8-6b5e9593e282 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:39.753903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c994c3-b987-472a-99f8-6b5e9593e282 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:39.753903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c994c3-b987-472a-99f8-6b5e9593e282 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:39.753903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c994c3-b987-472a-99f8-6b5e9593e282 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:39.754390 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c994c3-b987-472a-99f8-6b5e9593e282 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:39.754390 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c994c3-b987-472a-99f8-6b5e9593e282 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:39.771296 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1135/2283] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:39 2022] POST /identity/v3/users => generated 287 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:39.775790 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5e192f-6919-4df1-9b8e-0deb798b250b admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:39.787123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5e192f-6919-4df1-9b8e-0deb798b250b admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:39.792802 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aae4cd7-24fb-4d29-90c1-36011892fabc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:39.793549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae4cd7-24fb-4d29-90c1-36011892fabc admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:39.794144 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae4cd7-24fb-4d29-90c1-36011892fabc admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:39.794617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae4cd7-24fb-4d29-90c1-36011892fabc admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:39.795648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aae4cd7-24fb-4d29-90c1-36011892fabc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:39.797490 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aae4cd7-24fb-4d29-90c1-36011892fabc admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:39.809334 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1149/2284] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:39 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:39.815161 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c994c3-b987-472a-99f8-6b5e9593e282 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:39.826374 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c994c3-b987-472a-99f8-6b5e9593e282 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:39.832241 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d32d58-9738-4423-a4a9-f4c074f768d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:39.832874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d32d58-9738-4423-a4a9-f4c074f768d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:39.833195 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d32d58-9738-4423-a4a9-f4c074f768d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:39.833454 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d32d58-9738-4423-a4a9-f4c074f768d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:39.834749 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d32d58-9738-4423-a4a9-f4c074f768d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:39.837637 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d32d58-9738-4423-a4a9-f4c074f768d8 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:39.854348 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1136/2285] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:39 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:08:39.858698 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aae4cd7-24fb-4d29-90c1-36011892fabc admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:39.869880 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aae4cd7-24fb-4d29-90c1-36011892fabc admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:39.875329 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c8a1dc-ae0f-4110-8b38-f2130d5e101f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:39.876156 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c8a1dc-ae0f-4110-8b38-f2130d5e101f admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:39.876682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c8a1dc-ae0f-4110-8b38-f2130d5e101f admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:39.876997 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c8a1dc-ae0f-4110-8b38-f2130d5e101f admin admin] PATH_INFO: `/v3/projects/86480070152f464db7b9ee7f549819c3/users/80bb454bafaf4da2b5a85297be6ab37b/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:39.913011 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c8a1dc-ae0f-4110-8b38-f2130d5e101f admin admin] RBAC: Authorizing `identity:create_grant(project_id=86480070152f464db7b9ee7f549819c3, user_id=80bb454bafaf4da2b5a85297be6ab37b, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:39.915994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c8a1dc-ae0f-4110-8b38-f2130d5e101f admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:39.939997 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1150/2286] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:39 2022] PUT /identity/v3/projects/86480070152f464db7b9ee7f549819c3/users/80bb454bafaf4da2b5a85297be6ab37b/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:39.948985 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb9cb7b-b1c9-4056-a017-b7be016b44dc None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:39.950041 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb9cb7b-b1c9-4056-a017-b7be016b44dc None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:39.950670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb9cb7b-b1c9-4056-a017-b7be016b44dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:39.985013 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-acb9cb7b-b1c9-4056-a017-b7be016b44dc None None] MFA Rules not processed for user `80bb454bafaf4da2b5a85297be6ab37b`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:39.986815 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-acb9cb7b-b1c9-4056-a017-b7be016b44dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:40.093350 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1137/2287] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:39 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:40.097622 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c8a1dc-ae0f-4110-8b38-f2130d5e101f admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:40.191106 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c8a1dc-ae0f-4110-8b38-f2130d5e101f admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:40.195055 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cf7fd2-efc2-4b7d-9999-571baadb7114 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:40.195675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cf7fd2-efc2-4b7d-9999-571baadb7114 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:40.195978 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cf7fd2-efc2-4b7d-9999-571baadb7114 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:40.196248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cf7fd2-efc2-4b7d-9999-571baadb7114 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:40.196901 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cf7fd2-efc2-4b7d-9999-571baadb7114 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:40.198961 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cf7fd2-efc2-4b7d-9999-571baadb7114 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:40.244783 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1151/2288] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:40 2022] POST /identity/v3/projects => generated 373 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:08:40.246821 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb9cb7b-b1c9-4056-a017-b7be016b44dc None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:40.258213 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb9cb7b-b1c9-4056-a017-b7be016b44dc None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:40.295950 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:40.315953 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6263663f-fdd2-426b-a7cf-d071db1a7f87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:40.316562 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6263663f-fdd2-426b-a7cf-d071db1a7f87 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:40.316854 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6263663f-fdd2-426b-a7cf-d071db1a7f87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:40.317142 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6263663f-fdd2-426b-a7cf-d071db1a7f87 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:40.317770 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6263663f-fdd2-426b-a7cf-d071db1a7f87 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:40.319680 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6263663f-fdd2-426b-a7cf-d071db1a7f87 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:40.347408 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1138/2289] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:40 2022] POST /identity/v3/users => generated 293 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:40.351620 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cf7fd2-efc2-4b7d-9999-571baadb7114 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:40.362276 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cf7fd2-efc2-4b7d-9999-571baadb7114 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:40.366875 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b9f486-d574-41b1-98a6-88e23f8d92b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:40.367484 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b9f486-d574-41b1-98a6-88e23f8d92b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:40.367787 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b9f486-d574-41b1-98a6-88e23f8d92b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:40.368059 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b9f486-d574-41b1-98a6-88e23f8d92b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:40.368723 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83b9f486-d574-41b1-98a6-88e23f8d92b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:40.370316 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83b9f486-d574-41b1-98a6-88e23f8d92b0 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:40.381365 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1152/2290] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:40.385758 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6263663f-fdd2-426b-a7cf-d071db1a7f87 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:40.396441 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6263663f-fdd2-426b-a7cf-d071db1a7f87 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:40.401214 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2655b22e-a9d0-4027-a688-960d63f3f6a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:40.401874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2655b22e-a9d0-4027-a688-960d63f3f6a1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:40.402128 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2655b22e-a9d0-4027-a688-960d63f3f6a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:40.402386 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2655b22e-a9d0-4027-a688-960d63f3f6a1 admin admin] PATH_INFO: `/v3/projects/59daf8a0fe0d467faef9c239bf17949e/users/eb0d937733ed4e7588938f4b85f9ebe7/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:40.418347 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2655b22e-a9d0-4027-a688-960d63f3f6a1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=59daf8a0fe0d467faef9c239bf17949e, user_id=eb0d937733ed4e7588938f4b85f9ebe7, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:40.421281 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2655b22e-a9d0-4027-a688-960d63f3f6a1 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:40.437233 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1139/2291] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:40 2022] PUT /identity/v3/projects/59daf8a0fe0d467faef9c239bf17949e/users/eb0d937733ed4e7588938f4b85f9ebe7/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:40.441278 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b9f486-d574-41b1-98a6-88e23f8d92b0 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:40.452504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b9f486-d574-41b1-98a6-88e23f8d92b0 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:40.514063 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb30823-d6e5-4472-967d-6c51f396f60a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:40.514831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb30823-d6e5-4472-967d-6c51f396f60a admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:40.515202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb30823-d6e5-4472-967d-6c51f396f60a admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:40.515561 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb30823-d6e5-4472-967d-6c51f396f60a admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:40.516728 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb30823-d6e5-4472-967d-6c51f396f60a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:40.519115 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb30823-d6e5-4472-967d-6c51f396f60a admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:40.523888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:40.529765 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1153/2292] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:40 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:40.534413 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2655b22e-a9d0-4027-a688-960d63f3f6a1 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:40.547127 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2655b22e-a9d0-4027-a688-960d63f3f6a1 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:40.551764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067b2721-a32f-4c12-b934-43cdebc2af39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:40.552425 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067b2721-a32f-4c12-b934-43cdebc2af39 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:40.552701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067b2721-a32f-4c12-b934-43cdebc2af39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:40.552986 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067b2721-a32f-4c12-b934-43cdebc2af39 admin admin] PATH_INFO: `/v3/projects/59daf8a0fe0d467faef9c239bf17949e/users/eb0d937733ed4e7588938f4b85f9ebe7/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:40.555046 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067b2721-a32f-4c12-b934-43cdebc2af39 admin admin] RBAC: Authorizing `identity:create_grant(project_id=59daf8a0fe0d467faef9c239bf17949e, user_id=eb0d937733ed4e7588938f4b85f9ebe7, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:40.557994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067b2721-a32f-4c12-b934-43cdebc2af39 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:40.582791 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1140/2293] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:40 2022] PUT /identity/v3/projects/59daf8a0fe0d467faef9c239bf17949e/users/eb0d937733ed4e7588938f4b85f9ebe7/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:40.586756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8435257f-451f-43d9-8860-6ac992013a18 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:40.587199 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8435257f-451f-43d9-8860-6ac992013a18 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:40.587547 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8435257f-451f-43d9-8860-6ac992013a18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:40.619960 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-8435257f-451f-43d9-8860-6ac992013a18 None None] MFA Rules not processed for user `eb0d937733ed4e7588938f4b85f9ebe7`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:40.621291 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-8435257f-451f-43d9-8860-6ac992013a18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:40.694515 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1154/2294] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:40 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:08:40.706713 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067b2721-a32f-4c12-b934-43cdebc2af39 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:40.708549 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-067b2721-a32f-4c12-b934-43cdebc2af39 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:40.770035 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067b2721-a32f-4c12-b934-43cdebc2af39 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:40.773734 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f730673b-1537-4edd-9492-ba20c4788296 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:40.774415 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f730673b-1537-4edd-9492-ba20c4788296 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:40.774812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f730673b-1537-4edd-9492-ba20c4788296 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:40.775141 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f730673b-1537-4edd-9492-ba20c4788296 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:40.780092 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8435257f-451f-43d9-8860-6ac992013a18 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:40.781688 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-8435257f-451f-43d9-8860-6ac992013a18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:40.785797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f730673b-1537-4edd-9492-ba20c4788296 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:40.787241 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f730673b-1537-4edd-9492-ba20c4788296 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:40.796982 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1141/2295] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:40 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 01:08:40.860564 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8435257f-451f-43d9-8860-6ac992013a18 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:40.865128 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8968bd3-3b1e-4e16-a859-2b5e59a3e88d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:40.865757 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8968bd3-3b1e-4e16-a859-2b5e59a3e88d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:40.866204 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8968bd3-3b1e-4e16-a859-2b5e59a3e88d admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:40.866374 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8968bd3-3b1e-4e16-a859-2b5e59a3e88d admin admin] PATH_INFO: `/v3/users/d72f96923f3d45a099d2718a6af38cc4` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:40.867452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8968bd3-3b1e-4e16-a859-2b5e59a3e88d admin admin] RBAC: Authorizing `identity:delete_user(user_id=d72f96923f3d45a099d2718a6af38cc4)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:40.869529 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8968bd3-3b1e-4e16-a859-2b5e59a3e88d admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:40.936906 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b8968bd3-3b1e-4e16-a859-2b5e59a3e88d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd72f96923f3d45a099d2718a6af38cc4'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:40.948213 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b8968bd3-3b1e-4e16-a859-2b5e59a3e88d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd72f96923f3d45a099d2718a6af38cc4'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:40.948896 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b8968bd3-3b1e-4e16-a859-2b5e59a3e88d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd72f96923f3d45a099d2718a6af38cc4'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:40.963415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b8968bd3-3b1e-4e16-a859-2b5e59a3e88d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd72f96923f3d45a099d2718a6af38cc4'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:40.973728 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b8968bd3-3b1e-4e16-a859-2b5e59a3e88d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd72f96923f3d45a099d2718a6af38cc4'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:40.984025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1155/2296] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:40 2022] DELETE /identity/v3/users/d72f96923f3d45a099d2718a6af38cc4 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:40.985525 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e4026a-2495-4f4c-bb77-df48b0163b37 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:40.985809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e4026a-2495-4f4c-bb77-df48b0163b37 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:40.986076 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e4026a-2495-4f4c-bb77-df48b0163b37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:41.006152 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-64e4026a-2495-4f4c-bb77-df48b0163b37 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:41.007438 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-64e4026a-2495-4f4c-bb77-df48b0163b37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:41.083905 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1142/2297] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:41.091031 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8968bd3-3b1e-4e16-a859-2b5e59a3e88d admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:41.094209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-b8968bd3-3b1e-4e16-a859-2b5e59a3e88d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:41.103641 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:41.164750 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8968bd3-3b1e-4e16-a859-2b5e59a3e88d admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:41.168445 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802514e2-3965-4f0d-b5b5-66ec832d2a14 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:41.169119 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802514e2-3965-4f0d-b5b5-66ec832d2a14 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:41.169387 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802514e2-3965-4f0d-b5b5-66ec832d2a14 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:41.169714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802514e2-3965-4f0d-b5b5-66ec832d2a14 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:41.178830 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802514e2-3965-4f0d-b5b5-66ec832d2a14 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:41.180365 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802514e2-3965-4f0d-b5b5-66ec832d2a14 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:41.190054 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1156/2298] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 01:08:41.244071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e4026a-2495-4f4c-bb77-df48b0163b37 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:41.245538 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-64e4026a-2495-4f4c-bb77-df48b0163b37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:41.259897 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e4026a-2495-4f4c-bb77-df48b0163b37 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:41.265856 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1ac85b-1dfc-4369-a742-f84e8a51ef5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:41.266518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1ac85b-1dfc-4369-a742-f84e8a51ef5f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:41.266796 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1ac85b-1dfc-4369-a742-f84e8a51ef5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:41.267244 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1ac85b-1dfc-4369-a742-f84e8a51ef5f admin admin] PATH_INFO: `/v3/users/80bb454bafaf4da2b5a85297be6ab37b` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:41.268668 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1ac85b-1dfc-4369-a742-f84e8a51ef5f admin admin] RBAC: Authorizing `identity:delete_user(user_id=80bb454bafaf4da2b5a85297be6ab37b)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:41.271131 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1ac85b-1dfc-4369-a742-f84e8a51ef5f admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:41.315420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=90,keystone:INSERT=5,keystone:DELETE=5 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:41.332560 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-1e1ac85b-1dfc-4369-a742-f84e8a51ef5f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '80bb454bafaf4da2b5a85297be6ab37b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:41.341885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-1e1ac85b-1dfc-4369-a742-f84e8a51ef5f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '80bb454bafaf4da2b5a85297be6ab37b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:41.344634 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-1e1ac85b-1dfc-4369-a742-f84e8a51ef5f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '80bb454bafaf4da2b5a85297be6ab37b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:41.356511 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-1e1ac85b-1dfc-4369-a742-f84e8a51ef5f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '80bb454bafaf4da2b5a85297be6ab37b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:41.370114 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-1e1ac85b-1dfc-4369-a742-f84e8a51ef5f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '80bb454bafaf4da2b5a85297be6ab37b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:41.380145 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1143/2299] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:41 2022] DELETE /identity/v3/users/80bb454bafaf4da2b5a85297be6ab37b => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:41.384680 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045f226a-0d85-4a07-ab47-50f648a84408 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:41.384965 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045f226a-0d85-4a07-ab47-50f648a84408 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:41.385233 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045f226a-0d85-4a07-ab47-50f648a84408 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:41.410422 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-045f226a-0d85-4a07-ab47-50f648a84408 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:41.411675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-045f226a-0d85-4a07-ab47-50f648a84408 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:41.478779 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1157/2300] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:41.490804 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1ac85b-1dfc-4369-a742-f84e8a51ef5f admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:41.493437 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-1e1ac85b-1dfc-4369-a742-f84e8a51ef5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:41.571240 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1ac85b-1dfc-4369-a742-f84e8a51ef5f admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:41.576362 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe60131-149f-4419-ac07-a63fc0f11312 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:41.577131 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe60131-149f-4419-ac07-a63fc0f11312 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:41.577437 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe60131-149f-4419-ac07-a63fc0f11312 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:41.577732 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe60131-149f-4419-ac07-a63fc0f11312 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:41.588446 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe60131-149f-4419-ac07-a63fc0f11312 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:41.589912 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe60131-149f-4419-ac07-a63fc0f11312 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:41.601938 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1144/2301] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:41.785619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045f226a-0d85-4a07-ab47-50f648a84408 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:41.787325 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-045f226a-0d85-4a07-ab47-50f648a84408 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:41.803559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045f226a-0d85-4a07-ab47-50f648a84408 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:41.810327 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34c86b1-9d7d-4fcb-ad8c-b27ffa9260ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:41.810970 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34c86b1-9d7d-4fcb-ad8c-b27ffa9260ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:41.811238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34c86b1-9d7d-4fcb-ad8c-b27ffa9260ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:41.811700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34c86b1-9d7d-4fcb-ad8c-b27ffa9260ee admin admin] PATH_INFO: `/v3/projects/e419a44252d8478cb002db0af2e27bc4` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:41.813358 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34c86b1-9d7d-4fcb-ad8c-b27ffa9260ee admin admin] RBAC: Authorizing `identity:delete_project(project_id=e419a44252d8478cb002db0af2e27bc4)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:41.815269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34c86b1-9d7d-4fcb-ad8c-b27ffa9260ee admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:41.866937 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c34c86b1-9d7d-4fcb-ad8c-b27ffa9260ee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e419a44252d8478cb002db0af2e27bc4', 'request_id': 'req-c34c86b1-9d7d-4fcb-ad8c-b27ffa9260ee'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:41.875977 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c34c86b1-9d7d-4fcb-ad8c-b27ffa9260ee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e419a44252d8478cb002db0af2e27bc4', 'request_id': 'req-c34c86b1-9d7d-4fcb-ad8c-b27ffa9260ee'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:41.882053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c34c86b1-9d7d-4fcb-ad8c-b27ffa9260ee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e419a44252d8478cb002db0af2e27bc4', 'request_id': 'req-c34c86b1-9d7d-4fcb-ad8c-b27ffa9260ee'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:41.894572 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c34c86b1-9d7d-4fcb-ad8c-b27ffa9260ee admin admin] The token cache is being invalidate because project e419a44252d8478cb002db0af2e27bc4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:41.894885 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c34c86b1-9d7d-4fcb-ad8c-b27ffa9260ee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:41.895606 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c34c86b1-9d7d-4fcb-ad8c-b27ffa9260ee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:41.897256 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1158/2302] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:41 2022] DELETE /identity/v3/projects/e419a44252d8478cb002db0af2e27bc4 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:42.101560 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe60131-149f-4419-ac07-a63fc0f11312 service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:42.105009 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-6fe60131-149f-4419-ac07-a63fc0f11312 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:42.135078 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe60131-149f-4419-ac07-a63fc0f11312 service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:42.209694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cd95be-018d-4fb9-b617-53d04e9f4821 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:42.210659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cd95be-018d-4fb9-b617-53d04e9f4821 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:42.210933 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cd95be-018d-4fb9-b617-53d04e9f4821 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:42.211197 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cd95be-018d-4fb9-b617-53d04e9f4821 admin admin] PATH_INFO: `/v3/projects/86480070152f464db7b9ee7f549819c3` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:42.212490 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17cd95be-018d-4fb9-b617-53d04e9f4821 admin admin] RBAC: Authorizing `identity:delete_project(project_id=86480070152f464db7b9ee7f549819c3)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:42.214419 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17cd95be-018d-4fb9-b617-53d04e9f4821 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:42.263699 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-17cd95be-018d-4fb9-b617-53d04e9f4821 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '86480070152f464db7b9ee7f549819c3', 'request_id': 'req-17cd95be-018d-4fb9-b617-53d04e9f4821'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:42.265628 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:42.270453 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-17cd95be-018d-4fb9-b617-53d04e9f4821 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '86480070152f464db7b9ee7f549819c3', 'request_id': 'req-17cd95be-018d-4fb9-b617-53d04e9f4821'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:42.280597 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-17cd95be-018d-4fb9-b617-53d04e9f4821 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '86480070152f464db7b9ee7f549819c3', 'request_id': 'req-17cd95be-018d-4fb9-b617-53d04e9f4821'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:42.299884 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-17cd95be-018d-4fb9-b617-53d04e9f4821 admin admin] The token cache is being invalidate because project 86480070152f464db7b9ee7f549819c3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:42.299884 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-17cd95be-018d-4fb9-b617-53d04e9f4821 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:42.299884 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-17cd95be-018d-4fb9-b617-53d04e9f4821 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:42.299884 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1145/2303] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:42 2022] DELETE /identity/v3/projects/86480070152f464db7b9ee7f549819c3 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:42.300298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34c86b1-9d7d-4fcb-ad8c-b27ffa9260ee admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:42.300618 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c34c86b1-9d7d-4fcb-ad8c-b27ffa9260ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:42.313389 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34c86b1-9d7d-4fcb-ad8c-b27ffa9260ee admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:42.342588 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:42.374225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1de4260-a2e1-4b90-b3eb-a5f15fc03615 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:42.374866 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1de4260-a2e1-4b90-b3eb-a5f15fc03615 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:42.375319 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1de4260-a2e1-4b90-b3eb-a5f15fc03615 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:42.375589 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1de4260-a2e1-4b90-b3eb-a5f15fc03615 admin admin] PATH_INFO: `/v3/users/eb0d937733ed4e7588938f4b85f9ebe7` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:42.376784 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1de4260-a2e1-4b90-b3eb-a5f15fc03615 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eb0d937733ed4e7588938f4b85f9ebe7)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:42.378702 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1de4260-a2e1-4b90-b3eb-a5f15fc03615 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:42.433120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-d1de4260-a2e1-4b90-b3eb-a5f15fc03615 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eb0d937733ed4e7588938f4b85f9ebe7'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:42.442408 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-d1de4260-a2e1-4b90-b3eb-a5f15fc03615 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eb0d937733ed4e7588938f4b85f9ebe7'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:42.443025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-d1de4260-a2e1-4b90-b3eb-a5f15fc03615 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eb0d937733ed4e7588938f4b85f9ebe7'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:42.455084 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-d1de4260-a2e1-4b90-b3eb-a5f15fc03615 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eb0d937733ed4e7588938f4b85f9ebe7'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:42.462544 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-d1de4260-a2e1-4b90-b3eb-a5f15fc03615 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eb0d937733ed4e7588938f4b85f9ebe7'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:42.469412 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1159/2304] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:42 2022] DELETE /identity/v3/users/eb0d937733ed4e7588938f4b85f9ebe7 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:42.753818 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cd95be-018d-4fb9-b617-53d04e9f4821 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:42.755336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-17cd95be-018d-4fb9-b617-53d04e9f4821 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:42.768755 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cd95be-018d-4fb9-b617-53d04e9f4821 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:42.830395 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e156a3-0bd1-4a52-802f-b605dea0ac0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:42.831250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e156a3-0bd1-4a52-802f-b605dea0ac0a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:42.831523 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e156a3-0bd1-4a52-802f-b605dea0ac0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:42.831819 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e156a3-0bd1-4a52-802f-b605dea0ac0a admin admin] PATH_INFO: `/v3/projects/59daf8a0fe0d467faef9c239bf17949e` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:42.832953 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e156a3-0bd1-4a52-802f-b605dea0ac0a admin admin] RBAC: Authorizing `identity:delete_project(project_id=59daf8a0fe0d467faef9c239bf17949e)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:42.834860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e156a3-0bd1-4a52-802f-b605dea0ac0a admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:42.883420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-f0e156a3-0bd1-4a52-802f-b605dea0ac0a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '59daf8a0fe0d467faef9c239bf17949e', 'request_id': 'req-f0e156a3-0bd1-4a52-802f-b605dea0ac0a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:42.889356 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-f0e156a3-0bd1-4a52-802f-b605dea0ac0a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '59daf8a0fe0d467faef9c239bf17949e', 'request_id': 'req-f0e156a3-0bd1-4a52-802f-b605dea0ac0a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:42.900532 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-f0e156a3-0bd1-4a52-802f-b605dea0ac0a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '59daf8a0fe0d467faef9c239bf17949e', 'request_id': 'req-f0e156a3-0bd1-4a52-802f-b605dea0ac0a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:42.912556 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-f0e156a3-0bd1-4a52-802f-b605dea0ac0a admin admin] The token cache is being invalidate because project 59daf8a0fe0d467faef9c239bf17949e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:42.912834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-f0e156a3-0bd1-4a52-802f-b605dea0ac0a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:42.913450 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-f0e156a3-0bd1-4a52-802f-b605dea0ac0a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:42.914996 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1146/2305] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:42 2022] DELETE /identity/v3/projects/59daf8a0fe0d467faef9c239bf17949e => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:42.920308 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d6cae4-844c-4aba-926b-0126d486fdcc None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:42.920798 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d6cae4-844c-4aba-926b-0126d486fdcc None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:42.920798 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d6cae4-844c-4aba-926b-0126d486fdcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:42.941140 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-59d6cae4-844c-4aba-926b-0126d486fdcc None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:42.942316 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-59d6cae4-844c-4aba-926b-0126d486fdcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:43.004766 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1160/2306] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:43.012831 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c593c965-315f-480e-8dde-2048d01f3d8b None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:43.013114 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c593c965-315f-480e-8dde-2048d01f3d8b None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:43.013383 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c593c965-315f-480e-8dde-2048d01f3d8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:43.034249 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-c593c965-315f-480e-8dde-2048d01f3d8b None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:43.035449 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-c593c965-315f-480e-8dde-2048d01f3d8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:43.044533 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1147/2307] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:43.049336 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d6cae4-844c-4aba-926b-0126d486fdcc None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:43.059214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d6cae4-844c-4aba-926b-0126d486fdcc None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:43.064793 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec79ba6-2db8-41e8-8f8e-bb7b69e897f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:43.065387 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec79ba6-2db8-41e8-8f8e-bb7b69e897f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:43.065656 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec79ba6-2db8-41e8-8f8e-bb7b69e897f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:43.065918 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec79ba6-2db8-41e8-8f8e-bb7b69e897f0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:43.066552 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec79ba6-2db8-41e8-8f8e-bb7b69e897f0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:08:43.066846 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec79ba6-2db8-41e8-8f8e-bb7b69e897f0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:43.068491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec79ba6-2db8-41e8-8f8e-bb7b69e897f0 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:43.080167 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1161/2308] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:08:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 01:08:43.084724 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c593c965-315f-480e-8dde-2048d01f3d8b None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:43.094632 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c593c965-315f-480e-8dde-2048d01f3d8b None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:43.099576 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28955af3-7c76-492d-92d7-ae6d03dcdee7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:43.100215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28955af3-7c76-492d-92d7-ae6d03dcdee7 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:43.100483 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28955af3-7c76-492d-92d7-ae6d03dcdee7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:43.100749 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28955af3-7c76-492d-92d7-ae6d03dcdee7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:43.101386 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28955af3-7c76-492d-92d7-ae6d03dcdee7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:43.103291 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28955af3-7c76-492d-92d7-ae6d03dcdee7 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:43.134043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1148/2309] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:43 2022] POST /identity/v3/projects => generated 367 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:43.138585 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec79ba6-2db8-41e8-8f8e-bb7b69e897f0 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:43.148631 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec79ba6-2db8-41e8-8f8e-bb7b69e897f0 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:43.183200 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:43.212359 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c58a97c-2eec-4c3a-a2e0-c66113a735f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:43.212963 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c58a97c-2eec-4c3a-a2e0-c66113a735f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:43.213236 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c58a97c-2eec-4c3a-a2e0-c66113a735f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:43.213501 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c58a97c-2eec-4c3a-a2e0-c66113a735f1 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:43.214166 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c58a97c-2eec-4c3a-a2e0-c66113a735f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:43.216100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c58a97c-2eec-4c3a-a2e0-c66113a735f1 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:43.239001 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1162/2310] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:43 2022] POST /identity/v3/users => generated 284 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:43.243285 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28955af3-7c76-492d-92d7-ae6d03dcdee7 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:43.253774 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28955af3-7c76-492d-92d7-ae6d03dcdee7 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:43.258628 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429bdbb5-0b40-4304-a1b9-70ca74460191 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:43.259234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429bdbb5-0b40-4304-a1b9-70ca74460191 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:43.259506 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429bdbb5-0b40-4304-a1b9-70ca74460191 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:43.259799 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429bdbb5-0b40-4304-a1b9-70ca74460191 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:43.260438 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429bdbb5-0b40-4304-a1b9-70ca74460191 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:43.262029 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429bdbb5-0b40-4304-a1b9-70ca74460191 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:43.272491 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1149/2311] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:43 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:43.276497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c58a97c-2eec-4c3a-a2e0-c66113a735f1 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:43.287033 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c58a97c-2eec-4c3a-a2e0-c66113a735f1 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:43.292464 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29889994-0941-47b0-a6db-9e2955d8cab4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:43.293072 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29889994-0941-47b0-a6db-9e2955d8cab4 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:43.293343 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29889994-0941-47b0-a6db-9e2955d8cab4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:43.293631 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29889994-0941-47b0-a6db-9e2955d8cab4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:43.294270 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29889994-0941-47b0-a6db-9e2955d8cab4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:43.295988 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29889994-0941-47b0-a6db-9e2955d8cab4 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:43.306159 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1163/2312] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:43 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:43.310488 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429bdbb5-0b40-4304-a1b9-70ca74460191 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:43.320789 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429bdbb5-0b40-4304-a1b9-70ca74460191 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:43.325724 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99066493-2bd8-45af-b221-93669044a373 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:43.326489 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99066493-2bd8-45af-b221-93669044a373 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:43.326749 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99066493-2bd8-45af-b221-93669044a373 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:43.327022 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99066493-2bd8-45af-b221-93669044a373 admin admin] PATH_INFO: `/v3/projects/0e767348aa8e4a8fab1e6b7cb8bf71ee/users/a3c53944e4a649ed8a63479c947985a9/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:43.344972 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99066493-2bd8-45af-b221-93669044a373 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e767348aa8e4a8fab1e6b7cb8bf71ee, user_id=a3c53944e4a649ed8a63479c947985a9, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:43.347679 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99066493-2bd8-45af-b221-93669044a373 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:43.373630 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1150/2313] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:43 2022] PUT /identity/v3/projects/0e767348aa8e4a8fab1e6b7cb8bf71ee/users/a3c53944e4a649ed8a63479c947985a9/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:43.376473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f59bec-918d-4e0d-941b-0fb8d9a3a40e None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:43.376473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f59bec-918d-4e0d-941b-0fb8d9a3a40e None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:43.376473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f59bec-918d-4e0d-941b-0fb8d9a3a40e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:43.409304 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-95f59bec-918d-4e0d-941b-0fb8d9a3a40e None None] MFA Rules not processed for user `a3c53944e4a649ed8a63479c947985a9`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:43.410224 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-95f59bec-918d-4e0d-941b-0fb8d9a3a40e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:43.483605 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1164/2314] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:43 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:43.488325 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99066493-2bd8-45af-b221-93669044a373 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:43.498046 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:43.565984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99066493-2bd8-45af-b221-93669044a373 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:43.573835 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd66008-e2d2-4346-b3fa-7b511e058b50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:43.574695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd66008-e2d2-4346-b3fa-7b511e058b50 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:43.575046 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd66008-e2d2-4346-b3fa-7b511e058b50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:43.575382 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd66008-e2d2-4346-b3fa-7b511e058b50 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:43.576334 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd66008-e2d2-4346-b3fa-7b511e058b50 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:43.578374 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd66008-e2d2-4346-b3fa-7b511e058b50 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:43.607622 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1151/2315] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:43 2022] POST /identity/v3/projects => generated 367 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:43.611939 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f59bec-918d-4e0d-941b-0fb8d9a3a40e None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:43.622907 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f59bec-918d-4e0d-941b-0fb8d9a3a40e None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:43.685190 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e61e35-7666-47c2-a70f-72e844a7d35d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:43.685711 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e61e35-7666-47c2-a70f-72e844a7d35d admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:43.685985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e61e35-7666-47c2-a70f-72e844a7d35d admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:43.686264 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e61e35-7666-47c2-a70f-72e844a7d35d admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:43.687069 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e61e35-7666-47c2-a70f-72e844a7d35d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:43.689030 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e61e35-7666-47c2-a70f-72e844a7d35d admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:43.711933 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1165/2316] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:43 2022] POST /identity/v3/users => generated 290 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:43.721548 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd66008-e2d2-4346-b3fa-7b511e058b50 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:43.732758 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd66008-e2d2-4346-b3fa-7b511e058b50 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:43.738779 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007c0d77-ca82-4fa3-b6c7-67977ed30bea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:43.739378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007c0d77-ca82-4fa3-b6c7-67977ed30bea admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:43.739644 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007c0d77-ca82-4fa3-b6c7-67977ed30bea admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:43.739933 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007c0d77-ca82-4fa3-b6c7-67977ed30bea admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:43.740569 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007c0d77-ca82-4fa3-b6c7-67977ed30bea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:43.742161 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007c0d77-ca82-4fa3-b6c7-67977ed30bea admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:43.753114 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1152/2317] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:43 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:43.757528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e61e35-7666-47c2-a70f-72e844a7d35d admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:43.768455 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e61e35-7666-47c2-a70f-72e844a7d35d admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:43.773524 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f7e44f-c188-4d84-8f53-8feee020c3f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:43.774236 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f7e44f-c188-4d84-8f53-8feee020c3f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:43.774524 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f7e44f-c188-4d84-8f53-8feee020c3f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:43.774793 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f7e44f-c188-4d84-8f53-8feee020c3f4 admin admin] PATH_INFO: `/v3/projects/a22651620ce74a7eadc422b120fa9146/users/ac26ed6eb5de4c8fa35cd2ec700ce7ef/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:43.790177 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f7e44f-c188-4d84-8f53-8feee020c3f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a22651620ce74a7eadc422b120fa9146, user_id=ac26ed6eb5de4c8fa35cd2ec700ce7ef, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:43.792949 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f7e44f-c188-4d84-8f53-8feee020c3f4 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:43.808551 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1166/2318] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:43 2022] PUT /identity/v3/projects/a22651620ce74a7eadc422b120fa9146/users/ac26ed6eb5de4c8fa35cd2ec700ce7ef/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:43.812773 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007c0d77-ca82-4fa3-b6c7-67977ed30bea admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:43.823537 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007c0d77-ca82-4fa3-b6c7-67977ed30bea admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:43.884237 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e7786f-f829-4e3b-ab49-d339a6cc9f92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:43.884846 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e7786f-f829-4e3b-ab49-d339a6cc9f92 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:43.885120 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e7786f-f829-4e3b-ab49-d339a6cc9f92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:43.885386 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e7786f-f829-4e3b-ab49-d339a6cc9f92 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:43.886020 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e7786f-f829-4e3b-ab49-d339a6cc9f92 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:43.887653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e7786f-f829-4e3b-ab49-d339a6cc9f92 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:43.898542 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1153/2319] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:43 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:43.902867 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f7e44f-c188-4d84-8f53-8feee020c3f4 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:43.914588 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f7e44f-c188-4d84-8f53-8feee020c3f4 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:43.920272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27897ec-35ef-4ac9-a6c5-07c9947fdeeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:43.920928 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27897ec-35ef-4ac9-a6c5-07c9947fdeeb admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:43.921516 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27897ec-35ef-4ac9-a6c5-07c9947fdeeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:43.921516 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27897ec-35ef-4ac9-a6c5-07c9947fdeeb admin admin] PATH_INFO: `/v3/projects/a22651620ce74a7eadc422b120fa9146/users/ac26ed6eb5de4c8fa35cd2ec700ce7ef/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:43.923404 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27897ec-35ef-4ac9-a6c5-07c9947fdeeb admin admin] RBAC: Authorizing `identity:create_grant(project_id=a22651620ce74a7eadc422b120fa9146, user_id=ac26ed6eb5de4c8fa35cd2ec700ce7ef, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:43.926078 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27897ec-35ef-4ac9-a6c5-07c9947fdeeb admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:43.931697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:43.937795 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1167/2320] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:43 2022] PUT /identity/v3/projects/a22651620ce74a7eadc422b120fa9146/users/ac26ed6eb5de4c8fa35cd2ec700ce7ef/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:43.948575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63c0eec-67be-4139-941c-93c4e74c4e39 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:43.948857 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63c0eec-67be-4139-941c-93c4e74c4e39 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:43.949129 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63c0eec-67be-4139-941c-93c4e74c4e39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:43.977360 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-d63c0eec-67be-4139-941c-93c4e74c4e39 None None] MFA Rules not processed for user `ac26ed6eb5de4c8fa35cd2ec700ce7ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:43.978545 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d63c0eec-67be-4139-941c-93c4e74c4e39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:44.052338 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1154/2321] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:43 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:44.063474 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27897ec-35ef-4ac9-a6c5-07c9947fdeeb admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:44.065589 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-b27897ec-35ef-4ac9-a6c5-07c9947fdeeb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:44.137992 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27897ec-35ef-4ac9-a6c5-07c9947fdeeb admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:44.142043 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001bb649-2643-4b56-9351-8f812d1fb2c7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:44.142644 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001bb649-2643-4b56-9351-8f812d1fb2c7 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:44.142917 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001bb649-2643-4b56-9351-8f812d1fb2c7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:44.143187 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001bb649-2643-4b56-9351-8f812d1fb2c7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:44.153444 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001bb649-2643-4b56-9351-8f812d1fb2c7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:44.154876 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001bb649-2643-4b56-9351-8f812d1fb2c7 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:44.164607 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1168/2322] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:44 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:44.631431 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63c0eec-67be-4139-941c-93c4e74c4e39 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:44.632988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d63c0eec-67be-4139-941c-93c4e74c4e39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:44.638429 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:44.700958 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63c0eec-67be-4139-941c-93c4e74c4e39 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:44.704628 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a0a80c-6f8e-43c2-a121-c72a8acf391f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:44.705234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a0a80c-6f8e-43c2-a121-c72a8acf391f service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:44.705506 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a0a80c-6f8e-43c2-a121-c72a8acf391f service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:44.705912 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a0a80c-6f8e-43c2-a121-c72a8acf391f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:44.707634 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a3a0a80c-6f8e-43c2-a121-c72a8acf391f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:44.773604 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a0a80c-6f8e-43c2-a121-c72a8acf391f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:44.775051 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a0a80c-6f8e-43c2-a121-c72a8acf391f service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:44.785990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1155/2323] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:45.597192 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001bb649-2643-4b56-9351-8f812d1fb2c7 service cinder] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:45.607594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001bb649-2643-4b56-9351-8f812d1fb2c7 service cinder] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:45.612195 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cb1548-1f06-4c54-bb80-ba35d5290121 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:45.612818 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cb1548-1f06-4c54-bb80-ba35d5290121 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:45.613078 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cb1548-1f06-4c54-bb80-ba35d5290121 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:45.613329 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cb1548-1f06-4c54-bb80-ba35d5290121 admin admin] PATH_INFO: `/v3/users/a3c53944e4a649ed8a63479c947985a9` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:45.614417 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cb1548-1f06-4c54-bb80-ba35d5290121 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a3c53944e4a649ed8a63479c947985a9)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:45.616350 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cb1548-1f06-4c54-bb80-ba35d5290121 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:45.667101 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-33cb1548-1f06-4c54-bb80-ba35d5290121 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a3c53944e4a649ed8a63479c947985a9'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:45.680486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-33cb1548-1f06-4c54-bb80-ba35d5290121 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a3c53944e4a649ed8a63479c947985a9'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:45.681148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-33cb1548-1f06-4c54-bb80-ba35d5290121 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a3c53944e4a649ed8a63479c947985a9'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:45.696988 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-33cb1548-1f06-4c54-bb80-ba35d5290121 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a3c53944e4a649ed8a63479c947985a9'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:45.704888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-33cb1548-1f06-4c54-bb80-ba35d5290121 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a3c53944e4a649ed8a63479c947985a9'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:45.713238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1169/2324] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:45 2022] DELETE /identity/v3/users/a3c53944e4a649ed8a63479c947985a9 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:45.718001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec0b125-9932-4c9c-8c51-bc8bf6e8b4c5 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:45.718856 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec0b125-9932-4c9c-8c51-bc8bf6e8b4c5 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:45.719281 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec0b125-9932-4c9c-8c51-bc8bf6e8b4c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:45.740299 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-5ec0b125-9932-4c9c-8c51-bc8bf6e8b4c5 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:45.742085 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-5ec0b125-9932-4c9c-8c51-bc8bf6e8b4c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:45.813099 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1156/2325] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:45.823385 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cb1548-1f06-4c54-bb80-ba35d5290121 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:45.825107 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-33cb1548-1f06-4c54-bb80-ba35d5290121 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:45.888099 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cb1548-1f06-4c54-bb80-ba35d5290121 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:45.892963 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441a2434-0474-4626-b580-2b07b471914f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:45.893381 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441a2434-0474-4626-b580-2b07b471914f service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:45.893740 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441a2434-0474-4626-b580-2b07b471914f service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:45.894104 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441a2434-0474-4626-b580-2b07b471914f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:45.905786 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-441a2434-0474-4626-b580-2b07b471914f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:45.907408 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-441a2434-0474-4626-b580-2b07b471914f service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:45.917601 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1170/2326] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:46.576433 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec0b125-9932-4c9c-8c51-bc8bf6e8b4c5 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:46.579574 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-5ec0b125-9932-4c9c-8c51-bc8bf6e8b4c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:46.593253 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec0b125-9932-4c9c-8c51-bc8bf6e8b4c5 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:46.599160 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e2b423-50f0-423e-8dc1-1e9dd79b8733 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:46.600051 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e2b423-50f0-423e-8dc1-1e9dd79b8733 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:46.600479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e2b423-50f0-423e-8dc1-1e9dd79b8733 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:46.600798 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e2b423-50f0-423e-8dc1-1e9dd79b8733 admin admin] PATH_INFO: `/v3/projects/0e767348aa8e4a8fab1e6b7cb8bf71ee` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:46.602289 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e2b423-50f0-423e-8dc1-1e9dd79b8733 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0e767348aa8e4a8fab1e6b7cb8bf71ee)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:46.604075 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e2b423-50f0-423e-8dc1-1e9dd79b8733 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:46.659377 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-36e2b423-50f0-423e-8dc1-1e9dd79b8733 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0e767348aa8e4a8fab1e6b7cb8bf71ee', 'request_id': 'req-36e2b423-50f0-423e-8dc1-1e9dd79b8733'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:46.661638 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:46.666052 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-36e2b423-50f0-423e-8dc1-1e9dd79b8733 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0e767348aa8e4a8fab1e6b7cb8bf71ee', 'request_id': 'req-36e2b423-50f0-423e-8dc1-1e9dd79b8733'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:46.677573 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-36e2b423-50f0-423e-8dc1-1e9dd79b8733 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0e767348aa8e4a8fab1e6b7cb8bf71ee', 'request_id': 'req-36e2b423-50f0-423e-8dc1-1e9dd79b8733'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:46.692735 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-36e2b423-50f0-423e-8dc1-1e9dd79b8733 admin admin] The token cache is being invalidate because project 0e767348aa8e4a8fab1e6b7cb8bf71ee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:46.693142 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-36e2b423-50f0-423e-8dc1-1e9dd79b8733 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:46.693922 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-36e2b423-50f0-423e-8dc1-1e9dd79b8733 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:46.699695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1157/2327] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:46 2022] DELETE /identity/v3/projects/0e767348aa8e4a8fab1e6b7cb8bf71ee => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:46.701210 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441a2434-0474-4626-b580-2b07b471914f service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:46.712007 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441a2434-0474-4626-b580-2b07b471914f service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:46.725313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:46.787253 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee041462-3d53-4845-84ee-a417b2cd4ab7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:46.787936 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee041462-3d53-4845-84ee-a417b2cd4ab7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:46.788211 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee041462-3d53-4845-84ee-a417b2cd4ab7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:46.788481 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee041462-3d53-4845-84ee-a417b2cd4ab7 admin admin] PATH_INFO: `/v3/users/ac26ed6eb5de4c8fa35cd2ec700ce7ef` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:46.789928 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee041462-3d53-4845-84ee-a417b2cd4ab7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ac26ed6eb5de4c8fa35cd2ec700ce7ef)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:46.792755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee041462-3d53-4845-84ee-a417b2cd4ab7 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:46.859553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ee041462-3d53-4845-84ee-a417b2cd4ab7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ac26ed6eb5de4c8fa35cd2ec700ce7ef'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:46.872485 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ee041462-3d53-4845-84ee-a417b2cd4ab7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ac26ed6eb5de4c8fa35cd2ec700ce7ef'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:46.873159 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ee041462-3d53-4845-84ee-a417b2cd4ab7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ac26ed6eb5de4c8fa35cd2ec700ce7ef'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:46.887269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ee041462-3d53-4845-84ee-a417b2cd4ab7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ac26ed6eb5de4c8fa35cd2ec700ce7ef'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:46.899155 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ee041462-3d53-4845-84ee-a417b2cd4ab7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ac26ed6eb5de4c8fa35cd2ec700ce7ef'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:46.910781 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1171/2328] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:46 2022] DELETE /identity/v3/users/ac26ed6eb5de4c8fa35cd2ec700ce7ef => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:08:47.228101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e2b423-50f0-423e-8dc1-1e9dd79b8733 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:47.229608 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-36e2b423-50f0-423e-8dc1-1e9dd79b8733 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:47.247334 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e2b423-50f0-423e-8dc1-1e9dd79b8733 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:47.325173 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92df1da0-9dbb-4a25-98cb-0120ffca6656 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:47.325812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92df1da0-9dbb-4a25-98cb-0120ffca6656 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:47.326085 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92df1da0-9dbb-4a25-98cb-0120ffca6656 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:47.326354 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92df1da0-9dbb-4a25-98cb-0120ffca6656 admin admin] PATH_INFO: `/v3/projects/a22651620ce74a7eadc422b120fa9146` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:47.327543 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92df1da0-9dbb-4a25-98cb-0120ffca6656 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a22651620ce74a7eadc422b120fa9146)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:47.329701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92df1da0-9dbb-4a25-98cb-0120ffca6656 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:47.387405 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-92df1da0-9dbb-4a25-98cb-0120ffca6656 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a22651620ce74a7eadc422b120fa9146', 'request_id': 'req-92df1da0-9dbb-4a25-98cb-0120ffca6656'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:47.393824 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-92df1da0-9dbb-4a25-98cb-0120ffca6656 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a22651620ce74a7eadc422b120fa9146', 'request_id': 'req-92df1da0-9dbb-4a25-98cb-0120ffca6656'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:47.407142 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-92df1da0-9dbb-4a25-98cb-0120ffca6656 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a22651620ce74a7eadc422b120fa9146', 'request_id': 'req-92df1da0-9dbb-4a25-98cb-0120ffca6656'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:47.420272 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-92df1da0-9dbb-4a25-98cb-0120ffca6656 admin admin] The token cache is being invalidate because project a22651620ce74a7eadc422b120fa9146 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:47.420641 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-92df1da0-9dbb-4a25-98cb-0120ffca6656 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:47.421460 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-92df1da0-9dbb-4a25-98cb-0120ffca6656 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:47.427770 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1158/2329] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:47 2022] DELETE /identity/v3/projects/a22651620ce74a7eadc422b120fa9146 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:47.429046 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed9b112-ec04-42ca-8614-8a93dbc2dc37 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:47.429570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed9b112-ec04-42ca-8614-8a93dbc2dc37 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:47.429924 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed9b112-ec04-42ca-8614-8a93dbc2dc37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:47.452429 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-bed9b112-ec04-42ca-8614-8a93dbc2dc37 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:47.453839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-bed9b112-ec04-42ca-8614-8a93dbc2dc37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:47.516861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1172/2330] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:08:47.524643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45330b7f-4c5f-4ced-805d-19f51dba9f08 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:47.525000 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45330b7f-4c5f-4ced-805d-19f51dba9f08 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:47.525336 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45330b7f-4c5f-4ced-805d-19f51dba9f08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:47.546367 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-45330b7f-4c5f-4ced-805d-19f51dba9f08 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:47.547609 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-45330b7f-4c5f-4ced-805d-19f51dba9f08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:47.556873 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1159/2331] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:47.561511 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed9b112-ec04-42ca-8614-8a93dbc2dc37 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:47.571666 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed9b112-ec04-42ca-8614-8a93dbc2dc37 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:47.576306 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910564a0-ec87-4b8b-b679-d777ae2a2a27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:47.576932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910564a0-ec87-4b8b-b679-d777ae2a2a27 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:47.577203 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910564a0-ec87-4b8b-b679-d777ae2a2a27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:47.577473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910564a0-ec87-4b8b-b679-d777ae2a2a27 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:47.578221 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910564a0-ec87-4b8b-b679-d777ae2a2a27 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:08:47.578525 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910564a0-ec87-4b8b-b679-d777ae2a2a27 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:47.580155 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910564a0-ec87-4b8b-b679-d777ae2a2a27 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:47.592924 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1173/2332] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:08:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 01:08:47.597467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45330b7f-4c5f-4ced-805d-19f51dba9f08 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:47.606746 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45330b7f-4c5f-4ced-805d-19f51dba9f08 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:47.611638 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6f3c18-6fea-4b46-a393-f6873ec22ecd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:47.612275 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6f3c18-6fea-4b46-a393-f6873ec22ecd admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:47.612553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6f3c18-6fea-4b46-a393-f6873ec22ecd admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:47.612824 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6f3c18-6fea-4b46-a393-f6873ec22ecd admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:47.613455 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6f3c18-6fea-4b46-a393-f6873ec22ecd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:47.615358 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6f3c18-6fea-4b46-a393-f6873ec22ecd admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:47.649114 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1160/2333] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:47 2022] POST /identity/v3/projects => generated 385 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:08:47.650335 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910564a0-ec87-4b8b-b679-d777ae2a2a27 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:47.661016 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910564a0-ec87-4b8b-b679-d777ae2a2a27 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:47.682657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:47.727298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d33e48d-1dd2-47b9-9ff5-999b19fc5942 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:47.727941 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d33e48d-1dd2-47b9-9ff5-999b19fc5942 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:47.728260 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d33e48d-1dd2-47b9-9ff5-999b19fc5942 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:47.728582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d33e48d-1dd2-47b9-9ff5-999b19fc5942 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:47.729149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d33e48d-1dd2-47b9-9ff5-999b19fc5942 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:47.731117 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d33e48d-1dd2-47b9-9ff5-999b19fc5942 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:47.760339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1174/2334] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:47 2022] POST /identity/v3/users => generated 293 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:08:47.764347 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6f3c18-6fea-4b46-a393-f6873ec22ecd admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:47.777440 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6f3c18-6fea-4b46-a393-f6873ec22ecd admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:47.783667 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1323d3fa-f828-4e6b-a512-8b3c77102b75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:47.784520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1323d3fa-f828-4e6b-a512-8b3c77102b75 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:47.785418 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1323d3fa-f828-4e6b-a512-8b3c77102b75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:47.785888 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1323d3fa-f828-4e6b-a512-8b3c77102b75 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:47.787335 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1323d3fa-f828-4e6b-a512-8b3c77102b75 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:47.789348 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1323d3fa-f828-4e6b-a512-8b3c77102b75 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:47.802839 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1161/2335] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:47 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:08:47.807874 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d33e48d-1dd2-47b9-9ff5-999b19fc5942 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:47.820706 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d33e48d-1dd2-47b9-9ff5-999b19fc5942 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:47.826268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02cdd13-f477-45be-b0df-d9655267d831 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:47.826963 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02cdd13-f477-45be-b0df-d9655267d831 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:47.827372 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02cdd13-f477-45be-b0df-d9655267d831 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:47.827704 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02cdd13-f477-45be-b0df-d9655267d831 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:47.828524 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02cdd13-f477-45be-b0df-d9655267d831 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:47.830201 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02cdd13-f477-45be-b0df-d9655267d831 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:47.847019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1175/2336] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:47 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:47.848315 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1323d3fa-f828-4e6b-a512-8b3c77102b75 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:47.860219 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1323d3fa-f828-4e6b-a512-8b3c77102b75 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:47.865139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351f7c0a-414f-4bf5-9a5c-70121821fd38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:47.865892 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351f7c0a-414f-4bf5-9a5c-70121821fd38 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:47.866558 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351f7c0a-414f-4bf5-9a5c-70121821fd38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:47.866901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351f7c0a-414f-4bf5-9a5c-70121821fd38 admin admin] PATH_INFO: `/v3/projects/798f2dfe04504ad68fee6cd8461e7d18/users/c90cb7bc17f640918f2fdc3ed5cff7be/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:47.884153 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351f7c0a-414f-4bf5-9a5c-70121821fd38 admin admin] RBAC: Authorizing `identity:create_grant(project_id=798f2dfe04504ad68fee6cd8461e7d18, user_id=c90cb7bc17f640918f2fdc3ed5cff7be, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:47.887538 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351f7c0a-414f-4bf5-9a5c-70121821fd38 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:47.910989 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1162/2337] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:47 2022] PUT /identity/v3/projects/798f2dfe04504ad68fee6cd8461e7d18/users/c90cb7bc17f640918f2fdc3ed5cff7be/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:47.922890 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33dd9c84-c89f-4060-ab99-5e118d69f8ad None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:47.923169 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33dd9c84-c89f-4060-ab99-5e118d69f8ad None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:47.923823 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33dd9c84-c89f-4060-ab99-5e118d69f8ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:47.960811 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-33dd9c84-c89f-4060-ab99-5e118d69f8ad None None] MFA Rules not processed for user `c90cb7bc17f640918f2fdc3ed5cff7be`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:47.962041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-33dd9c84-c89f-4060-ab99-5e118d69f8ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:48.051161 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1176/2338] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:47 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:48.055681 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351f7c0a-414f-4bf5-9a5c-70121821fd38 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:48.060199 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:48.137033 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351f7c0a-414f-4bf5-9a5c-70121821fd38 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:48.142093 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874c3411-26b0-4f4f-8719-328df350b64b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:48.142711 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874c3411-26b0-4f4f-8719-328df350b64b admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:48.142987 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874c3411-26b0-4f4f-8719-328df350b64b admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:48.143256 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874c3411-26b0-4f4f-8719-328df350b64b admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:48.143921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874c3411-26b0-4f4f-8719-328df350b64b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:48.145919 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874c3411-26b0-4f4f-8719-328df350b64b admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:48.177724 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1163/2339] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:48 2022] POST /identity/v3/projects => generated 385 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:48.182319 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33dd9c84-c89f-4060-ab99-5e118d69f8ad None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:48.195302 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33dd9c84-c89f-4060-ab99-5e118d69f8ad None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:48.273216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31fd4c9-3627-4a11-9859-2f195d09ded9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:48.274054 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31fd4c9-3627-4a11-9859-2f195d09ded9 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:48.274386 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31fd4c9-3627-4a11-9859-2f195d09ded9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:48.274714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31fd4c9-3627-4a11-9859-2f195d09ded9 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:48.275483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b31fd4c9-3627-4a11-9859-2f195d09ded9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:48.277553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b31fd4c9-3627-4a11-9859-2f195d09ded9 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:48.315358 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1177/2340] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:48 2022] POST /identity/v3/users => generated 299 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:08:48.318021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874c3411-26b0-4f4f-8719-328df350b64b admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:48.334957 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31fd4c9-3627-4a11-9859-2f195d09ded9 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:48.334957 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-b31fd4c9-3627-4a11-9859-2f195d09ded9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:48.336860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874c3411-26b0-4f4f-8719-328df350b64b admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:48.343123 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024a8c04-bddd-412c-a105-2fcd35f458d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:48.344293 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024a8c04-bddd-412c-a105-2fcd35f458d2 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:48.344779 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024a8c04-bddd-412c-a105-2fcd35f458d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:48.345408 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024a8c04-bddd-412c-a105-2fcd35f458d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:48.346515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024a8c04-bddd-412c-a105-2fcd35f458d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:48.350353 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024a8c04-bddd-412c-a105-2fcd35f458d2 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:48.351438 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31fd4c9-3627-4a11-9859-2f195d09ded9 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:48.356316 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1a66c2-b0a5-4d73-a290-3e935089fc88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:48.356852 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1a66c2-b0a5-4d73-a290-3e935089fc88 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:48.357127 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1a66c2-b0a5-4d73-a290-3e935089fc88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:48.357395 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1a66c2-b0a5-4d73-a290-3e935089fc88 admin admin] PATH_INFO: `/v3/users/15e603f9898a430d8a53f21818e97722` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:48.358646 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1a66c2-b0a5-4d73-a290-3e935089fc88 admin admin] RBAC: Authorizing `identity:delete_user(user_id=15e603f9898a430d8a53f21818e97722)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:48.360543 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1a66c2-b0a5-4d73-a290-3e935089fc88 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:48.370941 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1164/2341] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:48 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:48.374743 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024a8c04-bddd-412c-a105-2fcd35f458d2 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:48.381642 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=93,keystone:INSERT=6,keystone:DELETE=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:48.389605 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024a8c04-bddd-412c-a105-2fcd35f458d2 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:48.394707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c50452c-9bd1-4788-9ca4-8e18876978e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:48.395567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c50452c-9bd1-4788-9ca4-8e18876978e8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:48.395702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c50452c-9bd1-4788-9ca4-8e18876978e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:48.396105 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c50452c-9bd1-4788-9ca4-8e18876978e8 admin admin] PATH_INFO: `/v3/projects/836228d79e894710857a44ab30fd2d65/users/cb1c4c01dd8d43fca6b0dc03d1f2f687/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:48.414165 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c50452c-9bd1-4788-9ca4-8e18876978e8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=836228d79e894710857a44ab30fd2d65, user_id=cb1c4c01dd8d43fca6b0dc03d1f2f687, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:48.417953 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c50452c-9bd1-4788-9ca4-8e18876978e8 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:48.427294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-4d1a66c2-b0a5-4d73-a290-3e935089fc88 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '15e603f9898a430d8a53f21818e97722'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:48.433912 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1165/2342] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:48 2022] PUT /identity/v3/projects/836228d79e894710857a44ab30fd2d65/users/cb1c4c01dd8d43fca6b0dc03d1f2f687/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:08:48.436098 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-4d1a66c2-b0a5-4d73-a290-3e935089fc88 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '15e603f9898a430d8a53f21818e97722'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:48.437003 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-4d1a66c2-b0a5-4d73-a290-3e935089fc88 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '15e603f9898a430d8a53f21818e97722'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:48.438488 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c50452c-9bd1-4788-9ca4-8e18876978e8 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:48.440597 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-0c50452c-9bd1-4788-9ca4-8e18876978e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:48.451489 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-4d1a66c2-b0a5-4d73-a290-3e935089fc88 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '15e603f9898a430d8a53f21818e97722'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:48.458521 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-4d1a66c2-b0a5-4d73-a290-3e935089fc88 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '15e603f9898a430d8a53f21818e97722'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:48.467756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1178/2343] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:48 2022] DELETE /identity/v3/users/15e603f9898a430d8a53f21818e97722 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:48.512515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c50452c-9bd1-4788-9ca4-8e18876978e8 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:48.571131 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7588896d-5ffa-4204-aea4-804e2e412747 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:48.571778 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7588896d-5ffa-4204-aea4-804e2e412747 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:48.572102 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7588896d-5ffa-4204-aea4-804e2e412747 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:48.572424 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7588896d-5ffa-4204-aea4-804e2e412747 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:48.573072 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7588896d-5ffa-4204-aea4-804e2e412747 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:48.574695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7588896d-5ffa-4204-aea4-804e2e412747 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:48.576502 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:48.587794 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1166/2344] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:48 2022] GET /identity/v3/roles => generated 1420 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:08:48.591121 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1a66c2-b0a5-4d73-a290-3e935089fc88 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:48.601606 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1a66c2-b0a5-4d73-a290-3e935089fc88 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:48.606312 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051bae7d-4800-45f3-aab3-2f25a48792a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:48.607045 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051bae7d-4800-45f3-aab3-2f25a48792a3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:48.607386 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051bae7d-4800-45f3-aab3-2f25a48792a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:48.607661 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051bae7d-4800-45f3-aab3-2f25a48792a3 admin admin] PATH_INFO: `/v3/projects/836228d79e894710857a44ab30fd2d65/users/cb1c4c01dd8d43fca6b0dc03d1f2f687/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:48.609946 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051bae7d-4800-45f3-aab3-2f25a48792a3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=836228d79e894710857a44ab30fd2d65, user_id=cb1c4c01dd8d43fca6b0dc03d1f2f687, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:48.612619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051bae7d-4800-45f3-aab3-2f25a48792a3 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:48.623447 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1179/2345] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:48 2022] PUT /identity/v3/projects/836228d79e894710857a44ab30fd2d65/users/cb1c4c01dd8d43fca6b0dc03d1f2f687/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:48.634282 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12febc59-2832-469f-aa16-0a9fe403a475 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:48.634589 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12febc59-2832-469f-aa16-0a9fe403a475 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:48.634966 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12febc59-2832-469f-aa16-0a9fe403a475 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:48.653051 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051bae7d-4800-45f3-aab3-2f25a48792a3 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:48.655179 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-051bae7d-4800-45f3-aab3-2f25a48792a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:48.669875 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-12febc59-2832-469f-aa16-0a9fe403a475 None None] MFA Rules not processed for user `cb1c4c01dd8d43fca6b0dc03d1f2f687`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:48.671098 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-12febc59-2832-469f-aa16-0a9fe403a475 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:48.725207 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051bae7d-4800-45f3-aab3-2f25a48792a3 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:48.730666 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89eefbf6-5fd8-4018-91e3-af5f41332751 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:48.731328 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eefbf6-5fd8-4018-91e3-af5f41332751 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:48.731605 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eefbf6-5fd8-4018-91e3-af5f41332751 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:48.731907 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eefbf6-5fd8-4018-91e3-af5f41332751 admin admin] PATH_INFO: `/v3/projects/a3bfc7112bb04760aa4e9adc50b12361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:48.733032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89eefbf6-5fd8-4018-91e3-af5f41332751 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a3bfc7112bb04760aa4e9adc50b12361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:48.735146 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89eefbf6-5fd8-4018-91e3-af5f41332751 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:48.743856 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1167/2346] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:48 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:48.755405 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12febc59-2832-469f-aa16-0a9fe403a475 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:48.756797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-12febc59-2832-469f-aa16-0a9fe403a475 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:48.788891 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-89eefbf6-5fd8-4018-91e3-af5f41332751 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a3bfc7112bb04760aa4e9adc50b12361', 'request_id': 'req-89eefbf6-5fd8-4018-91e3-af5f41332751'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:48.798883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-89eefbf6-5fd8-4018-91e3-af5f41332751 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a3bfc7112bb04760aa4e9adc50b12361', 'request_id': 'req-89eefbf6-5fd8-4018-91e3-af5f41332751'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:48.805095 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-89eefbf6-5fd8-4018-91e3-af5f41332751 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a3bfc7112bb04760aa4e9adc50b12361', 'request_id': 'req-89eefbf6-5fd8-4018-91e3-af5f41332751'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:48.817216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-89eefbf6-5fd8-4018-91e3-af5f41332751 admin admin] The token cache is being invalidate because project a3bfc7112bb04760aa4e9adc50b12361 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:48.817527 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-89eefbf6-5fd8-4018-91e3-af5f41332751 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:48.818172 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-89eefbf6-5fd8-4018-91e3-af5f41332751 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:48.819908 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1180/2347] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:48 2022] DELETE /identity/v3/projects/a3bfc7112bb04760aa4e9adc50b12361 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:48.824227 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12febc59-2832-469f-aa16-0a9fe403a475 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:48.824436 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59c32c0-d17e-426a-99eb-a80b4457b5f4 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:48.824717 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59c32c0-d17e-426a-99eb-a80b4457b5f4 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:48.824982 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59c32c0-d17e-426a-99eb-a80b4457b5f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:48.846457 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-e59c32c0-d17e-426a-99eb-a80b4457b5f4 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:48.847767 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e59c32c0-d17e-426a-99eb-a80b4457b5f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:48.852214 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=81,keystone:DELETE=16,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:48.876021 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc12019d-9a74-4f91-a78d-ed6bcf0cf14b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:48.876636 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc12019d-9a74-4f91-a78d-ed6bcf0cf14b service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:48.876910 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc12019d-9a74-4f91-a78d-ed6bcf0cf14b service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:48.877176 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc12019d-9a74-4f91-a78d-ed6bcf0cf14b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:48.878934 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-bc12019d-9a74-4f91-a78d-ed6bcf0cf14b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:48.910732 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1181/2348] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:08:48.919535 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7243cb57-25fd-4ad1-afcd-db54593641d9 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:48.919852 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7243cb57-25fd-4ad1-afcd-db54593641d9 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:48.920123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7243cb57-25fd-4ad1-afcd-db54593641d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:48.926213 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:48.941650 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc12019d-9a74-4f91-a78d-ed6bcf0cf14b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:48.942411 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-7243cb57-25fd-4ad1-afcd-db54593641d9 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:48.943097 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc12019d-9a74-4f91-a78d-ed6bcf0cf14b service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:48.943616 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-7243cb57-25fd-4ad1-afcd-db54593641d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:48.952578 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1182/2349] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:08:48.956329 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1168/2350] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:48 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 01:08:48.957210 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7243cb57-25fd-4ad1-afcd-db54593641d9 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:48.966822 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7243cb57-25fd-4ad1-afcd-db54593641d9 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:48.971801 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b93f33-a088-46d5-bbd8-64c331218409 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:48.972396 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b93f33-a088-46d5-bbd8-64c331218409 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:48.972888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b93f33-a088-46d5-bbd8-64c331218409 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:48.973154 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b93f33-a088-46d5-bbd8-64c331218409 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:48.974215 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b93f33-a088-46d5-bbd8-64c331218409 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:08:48.974549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b93f33-a088-46d5-bbd8-64c331218409 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:48.976230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b93f33-a088-46d5-bbd8-64c331218409 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:48.984211 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc12019d-9a74-4f91-a78d-ed6bcf0cf14b service cinder] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:48.985839 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-bc12019d-9a74-4f91-a78d-ed6bcf0cf14b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:48.988196 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1183/2351] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:08:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 01:08:48.992289 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b93f33-a088-46d5-bbd8-64c331218409 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:48.998834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc12019d-9a74-4f91-a78d-ed6bcf0cf14b service cinder] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:49.002427 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b93f33-a088-46d5-bbd8-64c331218409 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:49.003628 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd99ade8-91e8-434e-adb2-134b0c9672ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:49.004295 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd99ade8-91e8-434e-adb2-134b0c9672ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:49.004575 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd99ade8-91e8-434e-adb2-134b0c9672ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:49.004814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd99ade8-91e8-434e-adb2-134b0c9672ba admin admin] PATH_INFO: `/v3/users/c90cb7bc17f640918f2fdc3ed5cff7be` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:49.006065 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd99ade8-91e8-434e-adb2-134b0c9672ba admin admin] RBAC: Authorizing `identity:delete_user(user_id=c90cb7bc17f640918f2fdc3ed5cff7be)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:49.007063 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe6bb0e-a5c7-47bb-adcb-1fdd0cc7e5bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:49.007636 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe6bb0e-a5c7-47bb-adcb-1fdd0cc7e5bc admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:49.007942 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe6bb0e-a5c7-47bb-adcb-1fdd0cc7e5bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:49.008090 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd99ade8-91e8-434e-adb2-134b0c9672ba admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:49.008207 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe6bb0e-a5c7-47bb-adcb-1fdd0cc7e5bc admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:49.008858 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe6bb0e-a5c7-47bb-adcb-1fdd0cc7e5bc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:49.010703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe6bb0e-a5c7-47bb-adcb-1fdd0cc7e5bc admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:49.040143 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1184/2352] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:48 2022] POST /identity/v3/projects => generated 403 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:08:49.044140 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe6bb0e-a5c7-47bb-adcb-1fdd0cc7e5bc admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:49.060741 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-bd99ade8-91e8-434e-adb2-134b0c9672ba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c90cb7bc17f640918f2fdc3ed5cff7be'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:49.077379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-bd99ade8-91e8-434e-adb2-134b0c9672ba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c90cb7bc17f640918f2fdc3ed5cff7be'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:49.078675 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-bd99ade8-91e8-434e-adb2-134b0c9672ba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c90cb7bc17f640918f2fdc3ed5cff7be'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:49.087412 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-bd99ade8-91e8-434e-adb2-134b0c9672ba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c90cb7bc17f640918f2fdc3ed5cff7be'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:49.100753 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-bd99ade8-91e8-434e-adb2-134b0c9672ba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c90cb7bc17f640918f2fdc3ed5cff7be'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:49.106884 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1169/2353] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:48 2022] DELETE /identity/v3/users/c90cb7bc17f640918f2fdc3ed5cff7be => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:49.111490 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca196e8-ba6d-430f-9132-993ec011bec9 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:49.111791 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca196e8-ba6d-430f-9132-993ec011bec9 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:49.112394 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca196e8-ba6d-430f-9132-993ec011bec9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:49.122892 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe6bb0e-a5c7-47bb-adcb-1fdd0cc7e5bc admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:49.132604 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-9ca196e8-ba6d-430f-9132-993ec011bec9 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:49.133823 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-9ca196e8-ba6d-430f-9132-993ec011bec9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:49.178606 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39b1fc3-5d20-40fa-95ed-7e93635dd993 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:49.179129 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39b1fc3-5d20-40fa-95ed-7e93635dd993 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:49.179405 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39b1fc3-5d20-40fa-95ed-7e93635dd993 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:49.179674 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39b1fc3-5d20-40fa-95ed-7e93635dd993 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:49.180360 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39b1fc3-5d20-40fa-95ed-7e93635dd993 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:49.182280 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39b1fc3-5d20-40fa-95ed-7e93635dd993 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:49.193164 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1170/2354] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:08:49.204027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca196e8-ba6d-430f-9132-993ec011bec9 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:49.205512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-9ca196e8-ba6d-430f-9132-993ec011bec9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:49.209289 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1185/2355] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:49 2022] POST /identity/v3/users => generated 302 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:08:49.213712 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39b1fc3-5d20-40fa-95ed-7e93635dd993 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:49.215308 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-a39b1fc3-5d20-40fa-95ed-7e93635dd993 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:49.223417 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:49.227435 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39b1fc3-5d20-40fa-95ed-7e93635dd993 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:49.233523 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8bcf00-193b-424f-98dc-8b323b35bb59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:49.234229 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8bcf00-193b-424f-98dc-8b323b35bb59 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:49.234568 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8bcf00-193b-424f-98dc-8b323b35bb59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:49.234839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8bcf00-193b-424f-98dc-8b323b35bb59 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:49.235472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e8bcf00-193b-424f-98dc-8b323b35bb59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:49.237081 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e8bcf00-193b-424f-98dc-8b323b35bb59 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:49.247086 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1186/2356] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:49 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:08:49.251127 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8bcf00-193b-424f-98dc-8b323b35bb59 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:49.260783 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8bcf00-193b-424f-98dc-8b323b35bb59 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:49.263497 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca196e8-ba6d-430f-9132-993ec011bec9 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:49.265637 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b7e992-ba51-42dc-aaab-6703b4ff79b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:49.266464 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b7e992-ba51-42dc-aaab-6703b4ff79b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:49.266777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b7e992-ba51-42dc-aaab-6703b4ff79b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:49.267129 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b7e992-ba51-42dc-aaab-6703b4ff79b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:49.267419 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5e2422-746e-4d46-a191-491702479755 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:49.267784 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b7e992-ba51-42dc-aaab-6703b4ff79b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:49.268061 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5e2422-746e-4d46-a191-491702479755 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:49.268338 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5e2422-746e-4d46-a191-491702479755 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:49.268611 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5e2422-746e-4d46-a191-491702479755 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:49.269409 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b7e992-ba51-42dc-aaab-6703b4ff79b1 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:49.277847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5e2422-746e-4d46-a191-491702479755 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:49.279287 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5e2422-746e-4d46-a191-491702479755 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:49.280597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1187/2357] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:49 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:08:49.284897 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b7e992-ba51-42dc-aaab-6703b4ff79b1 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:49.289450 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1171/2358] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:49.295146 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b7e992-ba51-42dc-aaab-6703b4ff79b1 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:49.300315 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5709a270-75bd-4311-8212-4c1a03526a84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:49.301016 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5709a270-75bd-4311-8212-4c1a03526a84 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:49.301293 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5709a270-75bd-4311-8212-4c1a03526a84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:49.301635 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5709a270-75bd-4311-8212-4c1a03526a84 admin admin] PATH_INFO: `/v3/projects/fa7753acd74a4f3496e494990014e135/users/60b12f4219554fe2a1e4acbf3c8ff168/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:49.317090 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5709a270-75bd-4311-8212-4c1a03526a84 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa7753acd74a4f3496e494990014e135, user_id=60b12f4219554fe2a1e4acbf3c8ff168, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:49.319745 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5709a270-75bd-4311-8212-4c1a03526a84 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:49.335594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1188/2359] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:49 2022] PUT /identity/v3/projects/fa7753acd74a4f3496e494990014e135/users/60b12f4219554fe2a1e4acbf3c8ff168/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:49.341053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f011b87-71b2-4111-9f47-f59beefa386a None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:49.341605 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f011b87-71b2-4111-9f47-f59beefa386a None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:49.341605 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f011b87-71b2-4111-9f47-f59beefa386a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:49.350228 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:49.366396 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-8f011b87-71b2-4111-9f47-f59beefa386a None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:49.367614 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-8f011b87-71b2-4111-9f47-f59beefa386a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:49.429695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1172/2360] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:49.439876 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5709a270-75bd-4311-8212-4c1a03526a84 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:49.441737 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-5709a270-75bd-4311-8212-4c1a03526a84 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:49.502973 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5709a270-75bd-4311-8212-4c1a03526a84 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:49.506801 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c904f6b-5dda-4f92-ba71-47ea122a06a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:49.507385 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c904f6b-5dda-4f92-ba71-47ea122a06a5 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:49.507690 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c904f6b-5dda-4f92-ba71-47ea122a06a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:49.507966 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c904f6b-5dda-4f92-ba71-47ea122a06a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:49.517216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c904f6b-5dda-4f92-ba71-47ea122a06a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:49.518681 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c904f6b-5dda-4f92-ba71-47ea122a06a5 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:49.528304 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1189/2361] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 01:08:49.601396 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f011b87-71b2-4111-9f47-f59beefa386a None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:49.602955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-8f011b87-71b2-4111-9f47-f59beefa386a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:49.615763 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f011b87-71b2-4111-9f47-f59beefa386a None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:49.620883 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096503b8-6cc3-4006-97e7-f2bba1c3bd0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:49.621400 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096503b8-6cc3-4006-97e7-f2bba1c3bd0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:49.621679 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096503b8-6cc3-4006-97e7-f2bba1c3bd0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:49.621953 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096503b8-6cc3-4006-97e7-f2bba1c3bd0b admin admin] PATH_INFO: `/v3/projects/798f2dfe04504ad68fee6cd8461e7d18` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:49.623276 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096503b8-6cc3-4006-97e7-f2bba1c3bd0b admin admin] RBAC: Authorizing `identity:delete_project(project_id=798f2dfe04504ad68fee6cd8461e7d18)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:49.625457 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096503b8-6cc3-4006-97e7-f2bba1c3bd0b admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:49.685748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-096503b8-6cc3-4006-97e7-f2bba1c3bd0b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '798f2dfe04504ad68fee6cd8461e7d18', 'request_id': 'req-096503b8-6cc3-4006-97e7-f2bba1c3bd0b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:49.692259 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-096503b8-6cc3-4006-97e7-f2bba1c3bd0b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '798f2dfe04504ad68fee6cd8461e7d18', 'request_id': 'req-096503b8-6cc3-4006-97e7-f2bba1c3bd0b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:49.701100 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-096503b8-6cc3-4006-97e7-f2bba1c3bd0b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '798f2dfe04504ad68fee6cd8461e7d18', 'request_id': 'req-096503b8-6cc3-4006-97e7-f2bba1c3bd0b'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:49.714259 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-096503b8-6cc3-4006-97e7-f2bba1c3bd0b admin admin] The token cache is being invalidate because project 798f2dfe04504ad68fee6cd8461e7d18 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:49.714554 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-096503b8-6cc3-4006-97e7-f2bba1c3bd0b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:49.715633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-096503b8-6cc3-4006-97e7-f2bba1c3bd0b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:49.717669 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1173/2362] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:49 2022] DELETE /identity/v3/projects/798f2dfe04504ad68fee6cd8461e7d18 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:49.722065 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c904f6b-5dda-4f92-ba71-47ea122a06a5 service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:49.732631 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c904f6b-5dda-4f92-ba71-47ea122a06a5 service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:49.795226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2cf6ce-1a14-4f28-9014-4f78f78a1ba7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:49.795954 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2cf6ce-1a14-4f28-9014-4f78f78a1ba7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:49.796229 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2cf6ce-1a14-4f28-9014-4f78f78a1ba7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:49.796508 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2cf6ce-1a14-4f28-9014-4f78f78a1ba7 admin admin] PATH_INFO: `/v3/users/cb1c4c01dd8d43fca6b0dc03d1f2f687` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:49.797859 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2cf6ce-1a14-4f28-9014-4f78f78a1ba7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cb1c4c01dd8d43fca6b0dc03d1f2f687)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:49.799882 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2cf6ce-1a14-4f28-9014-4f78f78a1ba7 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:49.850819 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-1b2cf6ce-1a14-4f28-9014-4f78f78a1ba7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cb1c4c01dd8d43fca6b0dc03d1f2f687'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:49.860451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-1b2cf6ce-1a14-4f28-9014-4f78f78a1ba7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cb1c4c01dd8d43fca6b0dc03d1f2f687'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:49.861447 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-1b2cf6ce-1a14-4f28-9014-4f78f78a1ba7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cb1c4c01dd8d43fca6b0dc03d1f2f687'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:49.872760 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:INSERT=2,keystone:DELETE=9 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:49.891443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-1b2cf6ce-1a14-4f28-9014-4f78f78a1ba7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cb1c4c01dd8d43fca6b0dc03d1f2f687'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:49.901711 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-1b2cf6ce-1a14-4f28-9014-4f78f78a1ba7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cb1c4c01dd8d43fca6b0dc03d1f2f687'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:49.910797 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1190/2363] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:49 2022] DELETE /identity/v3/users/cb1c4c01dd8d43fca6b0dc03d1f2f687 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:50.236744 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096503b8-6cc3-4006-97e7-f2bba1c3bd0b admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:50.246383 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-096503b8-6cc3-4006-97e7-f2bba1c3bd0b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:50.268862 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9428308e-a2da-4365-9ec6-914d4c460e97 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:50.269145 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9428308e-a2da-4365-9ec6-914d4c460e97 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:50.269412 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9428308e-a2da-4365-9ec6-914d4c460e97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:50.275692 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096503b8-6cc3-4006-97e7-f2bba1c3bd0b admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:50.307468 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-9428308e-a2da-4365-9ec6-914d4c460e97 None None] MFA Rules not processed for user `60b12f4219554fe2a1e4acbf3c8ff168`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:50.308865 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-9428308e-a2da-4365-9ec6-914d4c460e97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:50.363147 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91599ff6-69fa-41bd-9e60-a25372389edd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:50.365116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91599ff6-69fa-41bd-9e60-a25372389edd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:50.365907 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91599ff6-69fa-41bd-9e60-a25372389edd admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:50.366369 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91599ff6-69fa-41bd-9e60-a25372389edd admin admin] PATH_INFO: `/v3/projects/836228d79e894710857a44ab30fd2d65` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:50.368386 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91599ff6-69fa-41bd-9e60-a25372389edd admin admin] RBAC: Authorizing `identity:delete_project(project_id=836228d79e894710857a44ab30fd2d65)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:50.371006 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91599ff6-69fa-41bd-9e60-a25372389edd admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:50.397782 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1191/2364] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:50 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:50.402394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9428308e-a2da-4365-9ec6-914d4c460e97 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:50.403854 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-9428308e-a2da-4365-9ec6-914d4c460e97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:50.417405 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:50.420560 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9428308e-a2da-4365-9ec6-914d4c460e97 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:50.442817 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-91599ff6-69fa-41bd-9e60-a25372389edd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '836228d79e894710857a44ab30fd2d65', 'request_id': 'req-91599ff6-69fa-41bd-9e60-a25372389edd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:50.450535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-91599ff6-69fa-41bd-9e60-a25372389edd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '836228d79e894710857a44ab30fd2d65', 'request_id': 'req-91599ff6-69fa-41bd-9e60-a25372389edd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:50.462399 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-91599ff6-69fa-41bd-9e60-a25372389edd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '836228d79e894710857a44ab30fd2d65', 'request_id': 'req-91599ff6-69fa-41bd-9e60-a25372389edd'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:50.477908 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-91599ff6-69fa-41bd-9e60-a25372389edd admin admin] The token cache is being invalidate because project 836228d79e894710857a44ab30fd2d65 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:50.478563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-91599ff6-69fa-41bd-9e60-a25372389edd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:50.479320 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-91599ff6-69fa-41bd-9e60-a25372389edd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:50.485636 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1174/2365] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:50 2022] DELETE /identity/v3/projects/836228d79e894710857a44ab30fd2d65 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:50.487376 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fac9c07-21ab-4c31-addd-2c96b98c5a6f None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:50.487739 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fac9c07-21ab-4c31-addd-2c96b98c5a6f None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:50.488090 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fac9c07-21ab-4c31-addd-2c96b98c5a6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:50.495015 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15be9a2-f9b9-424d-8c57-6b510fe36e64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:50.495668 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15be9a2-f9b9-424d-8c57-6b510fe36e64 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:50.495976 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15be9a2-f9b9-424d-8c57-6b510fe36e64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:50.496240 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15be9a2-f9b9-424d-8c57-6b510fe36e64 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:50.496881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15be9a2-f9b9-424d-8c57-6b510fe36e64 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:50.498804 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15be9a2-f9b9-424d-8c57-6b510fe36e64 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:50.512740 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-4fac9c07-21ab-4c31-addd-2c96b98c5a6f None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:50.513960 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-4fac9c07-21ab-4c31-addd-2c96b98c5a6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:50.522682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1175/2366] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:50.531086 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1160c3-3a37-4052-9e28-96f6ade3dc51 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:50.531299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1192/2367] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:50 2022] POST /identity/v3/projects => generated 403 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:08:50.531508 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1160c3-3a37-4052-9e28-96f6ade3dc51 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:50.531628 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1160c3-3a37-4052-9e28-96f6ade3dc51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:50.535106 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15be9a2-f9b9-424d-8c57-6b510fe36e64 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:50.537002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-f15be9a2-f9b9-424d-8c57-6b510fe36e64 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:50.552921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-3b1160c3-3a37-4052-9e28-96f6ade3dc51 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:50.554138 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-3b1160c3-3a37-4052-9e28-96f6ade3dc51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:50.600486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=98,keystone:DELETE=1,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:50.605265 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15be9a2-f9b9-424d-8c57-6b510fe36e64 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:50.611783 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41daec6b-5d92-41b5-82f0-fa3f0e06e6d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:50.612510 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41daec6b-5d92-41b5-82f0-fa3f0e06e6d6 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:50.613023 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41daec6b-5d92-41b5-82f0-fa3f0e06e6d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:50.613412 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41daec6b-5d92-41b5-82f0-fa3f0e06e6d6 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:50.614203 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41daec6b-5d92-41b5-82f0-fa3f0e06e6d6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:50.616396 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41daec6b-5d92-41b5-82f0-fa3f0e06e6d6 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:50.623007 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1176/2368] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:50.627946 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1160c3-3a37-4052-9e28-96f6ade3dc51 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:50.637516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1160c3-3a37-4052-9e28-96f6ade3dc51 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:50.642398 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90836c9-fbe3-4d7d-ad90-29e3b212a549 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:50.642575 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1193/2369] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:50 2022] POST /identity/v3/users => generated 308 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:08:50.642812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90836c9-fbe3-4d7d-ad90-29e3b212a549 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:50.643085 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90836c9-fbe3-4d7d-ad90-29e3b212a549 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:50.643348 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90836c9-fbe3-4d7d-ad90-29e3b212a549 admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:50.644146 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90836c9-fbe3-4d7d-ad90-29e3b212a549 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:08:50.644490 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90836c9-fbe3-4d7d-ad90-29e3b212a549 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:50.646103 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90836c9-fbe3-4d7d-ad90-29e3b212a549 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:50.646630 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41daec6b-5d92-41b5-82f0-fa3f0e06e6d6 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:50.657928 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41daec6b-5d92-41b5-82f0-fa3f0e06e6d6 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:50.658171 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1177/2370] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:08:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 01:08:50.662805 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90836c9-fbe3-4d7d-ad90-29e3b212a549 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:50.662963 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aaaaf76-b5f4-41c2-a88c-004593f126be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:50.663570 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aaaaf76-b5f4-41c2-a88c-004593f126be admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:50.663990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aaaaf76-b5f4-41c2-a88c-004593f126be admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:50.664261 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aaaaf76-b5f4-41c2-a88c-004593f126be admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:50.664898 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aaaaf76-b5f4-41c2-a88c-004593f126be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:50.666499 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aaaaf76-b5f4-41c2-a88c-004593f126be admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:50.673281 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90836c9-fbe3-4d7d-ad90-29e3b212a549 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:50.680865 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1194/2371] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:50 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:08:50.681024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789c8a7a-c35e-40ff-b174-3d7c88e3cfa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:50.681024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789c8a7a-c35e-40ff-b174-3d7c88e3cfa0 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:50.681024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789c8a7a-c35e-40ff-b174-3d7c88e3cfa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:50.681024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789c8a7a-c35e-40ff-b174-3d7c88e3cfa0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:50.681440 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789c8a7a-c35e-40ff-b174-3d7c88e3cfa0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:50.681924 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aaaaf76-b5f4-41c2-a88c-004593f126be admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:50.683066 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789c8a7a-c35e-40ff-b174-3d7c88e3cfa0 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:50.692026 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aaaaf76-b5f4-41c2-a88c-004593f126be admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:50.696662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681c5a6e-e83d-4975-b15c-111b1e05cb9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:50.697308 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681c5a6e-e83d-4975-b15c-111b1e05cb9a admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:50.697581 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681c5a6e-e83d-4975-b15c-111b1e05cb9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:50.697866 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681c5a6e-e83d-4975-b15c-111b1e05cb9a admin admin] PATH_INFO: `/v3/projects/efc84894339245c09f872c2049fbd498/users/29c92017fb8443c6b5d5de2d269226d8/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:50.712029 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1178/2372] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:50 2022] POST /identity/v3/projects => generated 387 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:08:50.713628 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681c5a6e-e83d-4975-b15c-111b1e05cb9a admin admin] RBAC: Authorizing `identity:create_grant(project_id=efc84894339245c09f872c2049fbd498, user_id=29c92017fb8443c6b5d5de2d269226d8, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:50.716194 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789c8a7a-c35e-40ff-b174-3d7c88e3cfa0 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:50.716482 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681c5a6e-e83d-4975-b15c-111b1e05cb9a admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:50.731971 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1195/2373] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:50 2022] PUT /identity/v3/projects/efc84894339245c09f872c2049fbd498/users/29c92017fb8443c6b5d5de2d269226d8/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:50.736175 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681c5a6e-e83d-4975-b15c-111b1e05cb9a admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:50.782135 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789c8a7a-c35e-40ff-b174-3d7c88e3cfa0 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:50.798618 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681c5a6e-e83d-4975-b15c-111b1e05cb9a admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:50.800105 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:50.804844 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124140ef-4207-4a66-a81c-45e3165b0d77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:50.805461 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124140ef-4207-4a66-a81c-45e3165b0d77 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:50.805743 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124140ef-4207-4a66-a81c-45e3165b0d77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:50.806004 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124140ef-4207-4a66-a81c-45e3165b0d77 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:50.806646 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124140ef-4207-4a66-a81c-45e3165b0d77 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:50.808294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124140ef-4207-4a66-a81c-45e3165b0d77 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:50.819209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1196/2374] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:50 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:08:50.823513 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124140ef-4207-4a66-a81c-45e3165b0d77 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:50.833487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124140ef-4207-4a66-a81c-45e3165b0d77 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:50.838683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5d2222-2b96-4f25-a033-4cffaad3a186 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:50.839365 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5d2222-2b96-4f25-a033-4cffaad3a186 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:50.839520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45db7499-8c81-4d11-8971-e5879b3857d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:50.839638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5d2222-2b96-4f25-a033-4cffaad3a186 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:50.839967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5d2222-2b96-4f25-a033-4cffaad3a186 admin admin] PATH_INFO: `/v3/projects/efc84894339245c09f872c2049fbd498/users/29c92017fb8443c6b5d5de2d269226d8/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:50.840164 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45db7499-8c81-4d11-8971-e5879b3857d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:50.840434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45db7499-8c81-4d11-8971-e5879b3857d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:50.840768 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45db7499-8c81-4d11-8971-e5879b3857d3 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:50.841321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45db7499-8c81-4d11-8971-e5879b3857d3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:50.841910 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5d2222-2b96-4f25-a033-4cffaad3a186 admin admin] RBAC: Authorizing `identity:create_grant(project_id=efc84894339245c09f872c2049fbd498, user_id=29c92017fb8443c6b5d5de2d269226d8, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:50.843227 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45db7499-8c81-4d11-8971-e5879b3857d3 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:50.844715 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5d2222-2b96-4f25-a033-4cffaad3a186 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:50.855899 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1197/2375] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:50 2022] PUT /identity/v3/projects/efc84894339245c09f872c2049fbd498/users/29c92017fb8443c6b5d5de2d269226d8/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:50.870708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1179/2376] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:50 2022] POST /identity/v3/users => generated 294 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:08:50.874642 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5d2222-2b96-4f25-a033-4cffaad3a186 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:50.942729 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5d2222-2b96-4f25-a033-4cffaad3a186 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:50.947660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ed49b0-6081-4ba0-a4c0-097d6c3db678 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:50.948301 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ed49b0-6081-4ba0-a4c0-097d6c3db678 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:50.948556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ed49b0-6081-4ba0-a4c0-097d6c3db678 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:50.948829 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ed49b0-6081-4ba0-a4c0-097d6c3db678 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:50.949522 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ed49b0-6081-4ba0-a4c0-097d6c3db678 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:50.951207 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ed49b0-6081-4ba0-a4c0-097d6c3db678 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:50.962453 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1198/2377] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:50 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:50.966703 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45db7499-8c81-4d11-8971-e5879b3857d3 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:50.977097 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45db7499-8c81-4d11-8971-e5879b3857d3 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:50.982437 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f7b2fe-dd67-44ca-be2c-c604190134e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:50.982975 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f7b2fe-dd67-44ca-be2c-c604190134e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:50.983245 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f7b2fe-dd67-44ca-be2c-c604190134e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:50.983506 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f7b2fe-dd67-44ca-be2c-c604190134e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:50.984439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f7b2fe-dd67-44ca-be2c-c604190134e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:50.986056 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f7b2fe-dd67-44ca-be2c-c604190134e5 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:50.997222 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1180/2378] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:50 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:08:51.001513 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ed49b0-6081-4ba0-a4c0-097d6c3db678 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:51.012533 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ed49b0-6081-4ba0-a4c0-097d6c3db678 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:51.017771 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351fe5fe-6573-4854-aca5-1c02ab082884 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:51.018451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351fe5fe-6573-4854-aca5-1c02ab082884 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:51.018724 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351fe5fe-6573-4854-aca5-1c02ab082884 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:51.018991 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351fe5fe-6573-4854-aca5-1c02ab082884 admin admin] PATH_INFO: `/v3/projects/e4915bad3f7b4ba4b5f4a2d816767676/users/5d0e89e9f724433aa4b2ded0a931a249/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:51.035362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351fe5fe-6573-4854-aca5-1c02ab082884 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e4915bad3f7b4ba4b5f4a2d816767676, user_id=5d0e89e9f724433aa4b2ded0a931a249, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:51.038383 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351fe5fe-6573-4854-aca5-1c02ab082884 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:51.053235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1199/2379] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:51 2022] PUT /identity/v3/projects/e4915bad3f7b4ba4b5f4a2d816767676/users/5d0e89e9f724433aa4b2ded0a931a249/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:51.064658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f18549e-736a-4e1a-be65-abbafe6d1546 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:51.065052 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f18549e-736a-4e1a-be65-abbafe6d1546 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:51.065783 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f18549e-736a-4e1a-be65-abbafe6d1546 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:51.097780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-0f18549e-736a-4e1a-be65-abbafe6d1546 None None] MFA Rules not processed for user `5d0e89e9f724433aa4b2ded0a931a249`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:51.099143 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-0f18549e-736a-4e1a-be65-abbafe6d1546 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:51.183321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1181/2380] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:51 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:51.187811 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351fe5fe-6573-4854-aca5-1c02ab082884 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:51.194188 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:51.265943 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351fe5fe-6573-4854-aca5-1c02ab082884 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:51.270789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c8ca7b-3c75-4a9a-803c-799eab328cc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:51.271392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c8ca7b-3c75-4a9a-803c-799eab328cc4 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:51.271666 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c8ca7b-3c75-4a9a-803c-799eab328cc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:51.271963 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c8ca7b-3c75-4a9a-803c-799eab328cc4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:51.272647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c8ca7b-3c75-4a9a-803c-799eab328cc4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:51.274565 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c8ca7b-3c75-4a9a-803c-799eab328cc4 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:51.302799 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1200/2381] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:51 2022] POST /identity/v3/projects => generated 387 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:51.306800 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f18549e-736a-4e1a-be65-abbafe6d1546 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:51.317461 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f18549e-736a-4e1a-be65-abbafe6d1546 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:51.380042 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08bea0b-25d5-4027-b0d1-425427c19cac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:51.380914 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08bea0b-25d5-4027-b0d1-425427c19cac admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:51.381186 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08bea0b-25d5-4027-b0d1-425427c19cac admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:51.381608 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08bea0b-25d5-4027-b0d1-425427c19cac admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:51.382194 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08bea0b-25d5-4027-b0d1-425427c19cac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:51.384236 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08bea0b-25d5-4027-b0d1-425427c19cac admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:51.406426 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1182/2382] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:51 2022] POST /identity/v3/users => generated 300 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:51.410792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c8ca7b-3c75-4a9a-803c-799eab328cc4 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:51.421635 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c8ca7b-3c75-4a9a-803c-799eab328cc4 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:51.426197 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be61cf2-165a-4bc3-af87-a589289bea89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:51.426812 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be61cf2-165a-4bc3-af87-a589289bea89 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:51.427082 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be61cf2-165a-4bc3-af87-a589289bea89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:51.427343 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be61cf2-165a-4bc3-af87-a589289bea89 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:51.428003 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be61cf2-165a-4bc3-af87-a589289bea89 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:51.429609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be61cf2-165a-4bc3-af87-a589289bea89 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:51.444732 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1201/2383] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:08:51.446182 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08bea0b-25d5-4027-b0d1-425427c19cac admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:51.457677 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08bea0b-25d5-4027-b0d1-425427c19cac admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:51.462841 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd9fe3c-fdbd-418e-9302-22444a8b082d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:51.463487 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd9fe3c-fdbd-418e-9302-22444a8b082d admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:51.463827 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd9fe3c-fdbd-418e-9302-22444a8b082d admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:51.464071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd9fe3c-fdbd-418e-9302-22444a8b082d admin admin] PATH_INFO: `/v3/projects/b10941322c694652a34272b41ce078d9/users/16d62859719d45ba83bbad39fa5f02b5/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:51.480181 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd9fe3c-fdbd-418e-9302-22444a8b082d admin admin] RBAC: Authorizing `identity:create_grant(project_id=b10941322c694652a34272b41ce078d9, user_id=16d62859719d45ba83bbad39fa5f02b5, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:51.482895 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd9fe3c-fdbd-418e-9302-22444a8b082d admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:51.486532 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:51.500080 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1183/2384] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:51 2022] PUT /identity/v3/projects/b10941322c694652a34272b41ce078d9/users/16d62859719d45ba83bbad39fa5f02b5/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:51.503962 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be61cf2-165a-4bc3-af87-a589289bea89 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:51.522488 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be61cf2-165a-4bc3-af87-a589289bea89 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:51.527066 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3402ce8-31e1-41eb-a932-6a753cf96d6e None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:51.527669 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3402ce8-31e1-41eb-a932-6a753cf96d6e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:51.528047 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3402ce8-31e1-41eb-a932-6a753cf96d6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:51.566916 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-f3402ce8-31e1-41eb-a932-6a753cf96d6e None None] MFA Rules not processed for user `29c92017fb8443c6b5d5de2d269226d8`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:51.568170 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-f3402ce8-31e1-41eb-a932-6a753cf96d6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:51.575528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58afbf1b-67af-43df-98c5-9d00bd97fe1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:51.576169 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58afbf1b-67af-43df-98c5-9d00bd97fe1b admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:51.576452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58afbf1b-67af-43df-98c5-9d00bd97fe1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:51.576773 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58afbf1b-67af-43df-98c5-9d00bd97fe1b admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:51.577413 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58afbf1b-67af-43df-98c5-9d00bd97fe1b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:51.579483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58afbf1b-67af-43df-98c5-9d00bd97fe1b admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:51.594883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1202/2385] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:51 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:08:51.596206 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58afbf1b-67af-43df-98c5-9d00bd97fe1b admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:51.610874 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58afbf1b-67af-43df-98c5-9d00bd97fe1b admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:51.617077 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95d14b8-bd81-4785-8557-1716593b15bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:51.618539 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95d14b8-bd81-4785-8557-1716593b15bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:51.619858 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95d14b8-bd81-4785-8557-1716593b15bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:51.620420 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95d14b8-bd81-4785-8557-1716593b15bb admin admin] PATH_INFO: `/v3/projects/b10941322c694652a34272b41ce078d9/users/16d62859719d45ba83bbad39fa5f02b5/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:51.622751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95d14b8-bd81-4785-8557-1716593b15bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=b10941322c694652a34272b41ce078d9, user_id=16d62859719d45ba83bbad39fa5f02b5, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:51.627101 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95d14b8-bd81-4785-8557-1716593b15bb admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:51.641812 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1203/2386] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:51 2022] PUT /identity/v3/projects/b10941322c694652a34272b41ce078d9/users/16d62859719d45ba83bbad39fa5f02b5/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:51.652599 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3258df7d-674d-42c2-a8ab-51c85b1137bd None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:51.653209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3258df7d-674d-42c2-a8ab-51c85b1137bd None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:51.653685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3258df7d-674d-42c2-a8ab-51c85b1137bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:51.657664 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1184/2387] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:51 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:51.687161 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-3258df7d-674d-42c2-a8ab-51c85b1137bd None None] MFA Rules not processed for user `16d62859719d45ba83bbad39fa5f02b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:51.688529 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3258df7d-674d-42c2-a8ab-51c85b1137bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:51.743410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:51.775216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1204/2388] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:51 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:51.785072 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3402ce8-31e1-41eb-a932-6a753cf96d6e None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:51.786701 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-f3402ce8-31e1-41eb-a932-6a753cf96d6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:51.863601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3402ce8-31e1-41eb-a932-6a753cf96d6e None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:51.917608 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e4b353-d22d-4883-a84c-5a43988d6d67 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:51.918219 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e4b353-d22d-4883-a84c-5a43988d6d67 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:51.918978 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e4b353-d22d-4883-a84c-5a43988d6d67 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:51.919261 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e4b353-d22d-4883-a84c-5a43988d6d67 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:51.929248 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e4b353-d22d-4883-a84c-5a43988d6d67 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:51.930695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e4b353-d22d-4883-a84c-5a43988d6d67 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:51.940733 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1185/2389] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:51 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:52.080197 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3258df7d-674d-42c2-a8ab-51c85b1137bd None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:52.144613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3258df7d-674d-42c2-a8ab-51c85b1137bd None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:52.149557 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f84eb3-d165-4b16-9b1b-ea299f449d00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:52.150402 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f84eb3-d165-4b16-9b1b-ea299f449d00 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:52.150961 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f84eb3-d165-4b16-9b1b-ea299f449d00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:52.151255 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f84eb3-d165-4b16-9b1b-ea299f449d00 admin admin] PATH_INFO: `/v3/users/5d0e89e9f724433aa4b2ded0a931a249` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:52.152398 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f84eb3-d165-4b16-9b1b-ea299f449d00 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5d0e89e9f724433aa4b2ded0a931a249)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:52.154307 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f84eb3-d165-4b16-9b1b-ea299f449d00 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:52.211247 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-53f84eb3-d165-4b16-9b1b-ea299f449d00 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5d0e89e9f724433aa4b2ded0a931a249'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:52.221984 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-53f84eb3-d165-4b16-9b1b-ea299f449d00 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5d0e89e9f724433aa4b2ded0a931a249'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:52.222878 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-53f84eb3-d165-4b16-9b1b-ea299f449d00 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5d0e89e9f724433aa4b2ded0a931a249'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:52.236469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-53f84eb3-d165-4b16-9b1b-ea299f449d00 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5d0e89e9f724433aa4b2ded0a931a249'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:52.243378 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-53f84eb3-d165-4b16-9b1b-ea299f449d00 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5d0e89e9f724433aa4b2ded0a931a249'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:52.250521 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1205/2390] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:52 2022] DELETE /identity/v3/users/5d0e89e9f724433aa4b2ded0a931a249 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:08:52.255103 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cfb07e-6bc2-4714-afa1-0aea2c0ec94a None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:52.255462 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cfb07e-6bc2-4714-afa1-0aea2c0ec94a None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:52.255760 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cfb07e-6bc2-4714-afa1-0aea2c0ec94a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:52.277792 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-47cfb07e-6bc2-4714-afa1-0aea2c0ec94a None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:52.278931 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-47cfb07e-6bc2-4714-afa1-0aea2c0ec94a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:52.315107 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:52.345330 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1186/2391] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:52.355980 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f84eb3-d165-4b16-9b1b-ea299f449d00 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:52.357404 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-53f84eb3-d165-4b16-9b1b-ea299f449d00 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:52.421101 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f84eb3-d165-4b16-9b1b-ea299f449d00 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:52.425749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470fbd30-e031-4481-8305-b4191d7e108b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:52.426487 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470fbd30-e031-4481-8305-b4191d7e108b service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:52.426763 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470fbd30-e031-4481-8305-b4191d7e108b service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:52.427030 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470fbd30-e031-4481-8305-b4191d7e108b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:52.436815 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470fbd30-e031-4481-8305-b4191d7e108b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:52.438614 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470fbd30-e031-4481-8305-b4191d7e108b service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:52.448558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1206/2392] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:52.737684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cfb07e-6bc2-4714-afa1-0aea2c0ec94a None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:52.741796 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-47cfb07e-6bc2-4714-afa1-0aea2c0ec94a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:52.757075 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cfb07e-6bc2-4714-afa1-0aea2c0ec94a None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:52.762046 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba4fa51-41cf-4ba7-9e79-377980c31b4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:52.762959 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba4fa51-41cf-4ba7-9e79-377980c31b4d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:52.763339 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba4fa51-41cf-4ba7-9e79-377980c31b4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:52.763683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470fbd30-e031-4481-8305-b4191d7e108b service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:52.764149 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba4fa51-41cf-4ba7-9e79-377980c31b4d admin admin] PATH_INFO: `/v3/projects/e4915bad3f7b4ba4b5f4a2d816767676` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:52.765149 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba4fa51-41cf-4ba7-9e79-377980c31b4d admin admin] RBAC: Authorizing `identity:delete_project(project_id=e4915bad3f7b4ba4b5f4a2d816767676)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:52.765364 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-470fbd30-e031-4481-8305-b4191d7e108b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:52.767217 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba4fa51-41cf-4ba7-9e79-377980c31b4d admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:52.784769 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470fbd30-e031-4481-8305-b4191d7e108b service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:52.790146 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cc43de-f9a5-457e-b56a-960c323ba0c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:52.791124 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cc43de-f9a5-457e-b56a-960c323ba0c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:52.791508 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cc43de-f9a5-457e-b56a-960c323ba0c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:52.791906 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cc43de-f9a5-457e-b56a-960c323ba0c8 admin admin] PATH_INFO: `/v3/users/60b12f4219554fe2a1e4acbf3c8ff168` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:52.794173 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6cc43de-f9a5-457e-b56a-960c323ba0c8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=60b12f4219554fe2a1e4acbf3c8ff168)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:52.796691 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6cc43de-f9a5-457e-b56a-960c323ba0c8 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:52.799643 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:52.819085 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-7ba4fa51-41cf-4ba7-9e79-377980c31b4d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e4915bad3f7b4ba4b5f4a2d816767676', 'request_id': 'req-7ba4fa51-41cf-4ba7-9e79-377980c31b4d'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:52.824750 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-7ba4fa51-41cf-4ba7-9e79-377980c31b4d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e4915bad3f7b4ba4b5f4a2d816767676', 'request_id': 'req-7ba4fa51-41cf-4ba7-9e79-377980c31b4d'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:52.833726 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-7ba4fa51-41cf-4ba7-9e79-377980c31b4d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e4915bad3f7b4ba4b5f4a2d816767676', 'request_id': 'req-7ba4fa51-41cf-4ba7-9e79-377980c31b4d'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:52.845580 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-7ba4fa51-41cf-4ba7-9e79-377980c31b4d admin admin] The token cache is being invalidate because project e4915bad3f7b4ba4b5f4a2d816767676 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:52.845871 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-7ba4fa51-41cf-4ba7-9e79-377980c31b4d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:52.846505 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-7ba4fa51-41cf-4ba7-9e79-377980c31b4d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:52.848056 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1187/2393] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:52 2022] DELETE /identity/v3/projects/e4915bad3f7b4ba4b5f4a2d816767676 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:52.849517 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c6cc43de-f9a5-457e-b56a-960c323ba0c8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '60b12f4219554fe2a1e4acbf3c8ff168'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:52.852001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba4fa51-41cf-4ba7-9e79-377980c31b4d admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:52.853352 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-7ba4fa51-41cf-4ba7-9e79-377980c31b4d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:52.859016 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c6cc43de-f9a5-457e-b56a-960c323ba0c8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '60b12f4219554fe2a1e4acbf3c8ff168'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:52.859796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c6cc43de-f9a5-457e-b56a-960c323ba0c8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '60b12f4219554fe2a1e4acbf3c8ff168'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:52.872779 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c6cc43de-f9a5-457e-b56a-960c323ba0c8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '60b12f4219554fe2a1e4acbf3c8ff168'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:52.880123 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c6cc43de-f9a5-457e-b56a-960c323ba0c8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '60b12f4219554fe2a1e4acbf3c8ff168'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:52.887205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1207/2394] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:52 2022] DELETE /identity/v3/users/60b12f4219554fe2a1e4acbf3c8ff168 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:08:52.919042 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba4fa51-41cf-4ba7-9e79-377980c31b4d admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:52.977561 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bbb5f2-0697-4196-94f9-f784dfb61327 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:52.978243 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bbb5f2-0697-4196-94f9-f784dfb61327 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:52.978489 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bbb5f2-0697-4196-94f9-f784dfb61327 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:52.978764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bbb5f2-0697-4196-94f9-f784dfb61327 admin admin] PATH_INFO: `/v3/users/16d62859719d45ba83bbad39fa5f02b5` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:52.980017 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bbb5f2-0697-4196-94f9-f784dfb61327 admin admin] RBAC: Authorizing `identity:delete_user(user_id=16d62859719d45ba83bbad39fa5f02b5)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:52.981949 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bbb5f2-0697-4196-94f9-f784dfb61327 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:52.998738 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:53.037093 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-f6bbb5f2-0697-4196-94f9-f784dfb61327 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '16d62859719d45ba83bbad39fa5f02b5'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.047321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-f6bbb5f2-0697-4196-94f9-f784dfb61327 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '16d62859719d45ba83bbad39fa5f02b5'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.048415 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-f6bbb5f2-0697-4196-94f9-f784dfb61327 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '16d62859719d45ba83bbad39fa5f02b5'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.058398 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-f6bbb5f2-0697-4196-94f9-f784dfb61327 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '16d62859719d45ba83bbad39fa5f02b5'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.072288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cc43de-f9a5-457e-b56a-960c323ba0c8 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:53.074445 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c6cc43de-f9a5-457e-b56a-960c323ba0c8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:53.074852 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-f6bbb5f2-0697-4196-94f9-f784dfb61327 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '16d62859719d45ba83bbad39fa5f02b5'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.087780 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1188/2395] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:52 2022] DELETE /identity/v3/users/16d62859719d45ba83bbad39fa5f02b5 => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:08:53.088917 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cc43de-f9a5-457e-b56a-960c323ba0c8 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:53.147004 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b25c248-5c67-43f3-a233-1a080ec888c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:53.147669 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b25c248-5c67-43f3-a233-1a080ec888c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:53.147970 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b25c248-5c67-43f3-a233-1a080ec888c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:53.148282 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b25c248-5c67-43f3-a233-1a080ec888c0 admin admin] PATH_INFO: `/v3/projects/fa7753acd74a4f3496e494990014e135` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:53.149462 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b25c248-5c67-43f3-a233-1a080ec888c0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa7753acd74a4f3496e494990014e135)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:53.151534 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b25c248-5c67-43f3-a233-1a080ec888c0 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:53.203810 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-5b25c248-5c67-43f3-a233-1a080ec888c0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa7753acd74a4f3496e494990014e135', 'request_id': 'req-5b25c248-5c67-43f3-a233-1a080ec888c0'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.213314 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-5b25c248-5c67-43f3-a233-1a080ec888c0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa7753acd74a4f3496e494990014e135', 'request_id': 'req-5b25c248-5c67-43f3-a233-1a080ec888c0'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.219848 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-5b25c248-5c67-43f3-a233-1a080ec888c0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa7753acd74a4f3496e494990014e135', 'request_id': 'req-5b25c248-5c67-43f3-a233-1a080ec888c0'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.231475 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-5b25c248-5c67-43f3-a233-1a080ec888c0 admin admin] The token cache is being invalidate because project fa7753acd74a4f3496e494990014e135 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:53.231816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-5b25c248-5c67-43f3-a233-1a080ec888c0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.232503 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-5b25c248-5c67-43f3-a233-1a080ec888c0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.234211 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1208/2396] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:53 2022] DELETE /identity/v3/projects/fa7753acd74a4f3496e494990014e135 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:53.238524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bbb5f2-0697-4196-94f9-f784dfb61327 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:53.240349 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-f6bbb5f2-0697-4196-94f9-f784dfb61327 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:53.254659 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bbb5f2-0697-4196-94f9-f784dfb61327 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:53.314061 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a98f0b-eebd-40b9-97ea-aa9ff8fa5e50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:53.314778 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a98f0b-eebd-40b9-97ea-aa9ff8fa5e50 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:53.315118 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a98f0b-eebd-40b9-97ea-aa9ff8fa5e50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:53.315455 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a98f0b-eebd-40b9-97ea-aa9ff8fa5e50 admin admin] PATH_INFO: `/v3/users/29c92017fb8443c6b5d5de2d269226d8` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:53.316604 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a98f0b-eebd-40b9-97ea-aa9ff8fa5e50 admin admin] RBAC: Authorizing `identity:delete_user(user_id=29c92017fb8443c6b5d5de2d269226d8)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:53.318812 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a98f0b-eebd-40b9-97ea-aa9ff8fa5e50 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:53.371660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-56a98f0b-eebd-40b9-97ea-aa9ff8fa5e50 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '29c92017fb8443c6b5d5de2d269226d8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.381104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-56a98f0b-eebd-40b9-97ea-aa9ff8fa5e50 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '29c92017fb8443c6b5d5de2d269226d8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.381820 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-56a98f0b-eebd-40b9-97ea-aa9ff8fa5e50 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '29c92017fb8443c6b5d5de2d269226d8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.392781 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b25c248-5c67-43f3-a233-1a080ec888c0 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:53.394040 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-56a98f0b-eebd-40b9-97ea-aa9ff8fa5e50 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '29c92017fb8443c6b5d5de2d269226d8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.394430 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-5b25c248-5c67-43f3-a233-1a080ec888c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:53.408765 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b25c248-5c67-43f3-a233-1a080ec888c0 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:53.410638 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-56a98f0b-eebd-40b9-97ea-aa9ff8fa5e50 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '29c92017fb8443c6b5d5de2d269226d8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.413913 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58fa585-e629-4cd8-8f31-71809d068eae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:53.414670 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58fa585-e629-4cd8-8f31-71809d068eae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:53.415076 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58fa585-e629-4cd8-8f31-71809d068eae admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:53.415473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58fa585-e629-4cd8-8f31-71809d068eae admin admin] PATH_INFO: `/v3/projects/b10941322c694652a34272b41ce078d9` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:53.416987 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58fa585-e629-4cd8-8f31-71809d068eae admin admin] RBAC: Authorizing `identity:delete_project(project_id=b10941322c694652a34272b41ce078d9)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:53.417414 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1189/2397] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:53 2022] DELETE /identity/v3/users/29c92017fb8443c6b5d5de2d269226d8 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:53.419249 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58fa585-e629-4cd8-8f31-71809d068eae admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:53.427647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:53.474340 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-f58fa585-e629-4cd8-8f31-71809d068eae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b10941322c694652a34272b41ce078d9', 'request_id': 'req-f58fa585-e629-4cd8-8f31-71809d068eae'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.482442 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-f58fa585-e629-4cd8-8f31-71809d068eae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b10941322c694652a34272b41ce078d9', 'request_id': 'req-f58fa585-e629-4cd8-8f31-71809d068eae'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.488758 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-f58fa585-e629-4cd8-8f31-71809d068eae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b10941322c694652a34272b41ce078d9', 'request_id': 'req-f58fa585-e629-4cd8-8f31-71809d068eae'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.501375 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-f58fa585-e629-4cd8-8f31-71809d068eae admin admin] The token cache is being invalidate because project b10941322c694652a34272b41ce078d9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:53.501656 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-f58fa585-e629-4cd8-8f31-71809d068eae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.502328 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-f58fa585-e629-4cd8-8f31-71809d068eae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.503912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1209/2398] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:53 2022] DELETE /identity/v3/projects/b10941322c694652a34272b41ce078d9 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:08:53.508957 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9227850f-dc55-4dd5-8884-9cd0dae1bd23 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:53.509361 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9227850f-dc55-4dd5-8884-9cd0dae1bd23 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:53.509692 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9227850f-dc55-4dd5-8884-9cd0dae1bd23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:53.530112 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-9227850f-dc55-4dd5-8884-9cd0dae1bd23 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:53.531741 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-9227850f-dc55-4dd5-8884-9cd0dae1bd23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:53.536796 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:53.597071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1190/2399] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:53.606949 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58fa585-e629-4cd8-8f31-71809d068eae admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:53.608361 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-f58fa585-e629-4cd8-8f31-71809d068eae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:53.625796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58fa585-e629-4cd8-8f31-71809d068eae admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:53.631685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da1a039-3bbb-44ef-9615-388d09947862 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:53.632572 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da1a039-3bbb-44ef-9615-388d09947862 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:53.632852 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da1a039-3bbb-44ef-9615-388d09947862 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:53.633125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da1a039-3bbb-44ef-9615-388d09947862 admin admin] PATH_INFO: `/v3/projects/efc84894339245c09f872c2049fbd498` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:53.634355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da1a039-3bbb-44ef-9615-388d09947862 admin admin] RBAC: Authorizing `identity:delete_project(project_id=efc84894339245c09f872c2049fbd498)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:53.637205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da1a039-3bbb-44ef-9615-388d09947862 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:53.678796 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ebb258-2160-41b3-8bb0-12cf68681f3c None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:53.679144 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ebb258-2160-41b3-8bb0-12cf68681f3c None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:53.679508 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ebb258-2160-41b3-8bb0-12cf68681f3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:53.694499 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-9da1a039-3bbb-44ef-9615-388d09947862 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'efc84894339245c09f872c2049fbd498', 'request_id': 'req-9da1a039-3bbb-44ef-9615-388d09947862'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.700612 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-76ebb258-2160-41b3-8bb0-12cf68681f3c None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:53.701874 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-76ebb258-2160-41b3-8bb0-12cf68681f3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:53.703967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-9da1a039-3bbb-44ef-9615-388d09947862 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'efc84894339245c09f872c2049fbd498', 'request_id': 'req-9da1a039-3bbb-44ef-9615-388d09947862'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.710986 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-9da1a039-3bbb-44ef-9615-388d09947862 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'efc84894339245c09f872c2049fbd498', 'request_id': 'req-9da1a039-3bbb-44ef-9615-388d09947862'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.723555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-9da1a039-3bbb-44ef-9615-388d09947862 admin admin] The token cache is being invalidate because project efc84894339245c09f872c2049fbd498 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:53.723991 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-9da1a039-3bbb-44ef-9615-388d09947862 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.724767 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-9da1a039-3bbb-44ef-9615-388d09947862 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:53.726293 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1210/2400] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:53 2022] DELETE /identity/v3/projects/efc84894339245c09f872c2049fbd498 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:53.731075 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce07c8d2-2d0d-4cc2-9c2b-4e8e532f762c None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:53.731352 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce07c8d2-2d0d-4cc2-9c2b-4e8e532f762c None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:53.731611 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce07c8d2-2d0d-4cc2-9c2b-4e8e532f762c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:53.751934 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-ce07c8d2-2d0d-4cc2-9c2b-4e8e532f762c None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:53.753387 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ce07c8d2-2d0d-4cc2-9c2b-4e8e532f762c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:53.769578 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1191/2401] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:08:53.773994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ebb258-2160-41b3-8bb0-12cf68681f3c None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:53.783427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ebb258-2160-41b3-8bb0-12cf68681f3c None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:53.788799 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbd660e-b56c-49e9-98cd-2aa39d86ce8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:53.789407 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbd660e-b56c-49e9-98cd-2aa39d86ce8c admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:53.789670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbd660e-b56c-49e9-98cd-2aa39d86ce8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:53.789937 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbd660e-b56c-49e9-98cd-2aa39d86ce8c admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:53.790579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbd660e-b56c-49e9-98cd-2aa39d86ce8c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:08:53.790877 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbd660e-b56c-49e9-98cd-2aa39d86ce8c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:53.792532 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbd660e-b56c-49e9-98cd-2aa39d86ce8c admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:53.808527 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1192/2402] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:08:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 01:08:53.809516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbd660e-b56c-49e9-98cd-2aa39d86ce8c admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:53.815329 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1211/2403] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:08:53.819367 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbd660e-b56c-49e9-98cd-2aa39d86ce8c admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:53.822945 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1488873e-eaa1-481e-9d8a-7a86759e91b6 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:53.823363 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1488873e-eaa1-481e-9d8a-7a86759e91b6 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:53.823658 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1488873e-eaa1-481e-9d8a-7a86759e91b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:53.824271 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c4e77a-6bdd-41a4-8a57-b7eb6af064f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:53.824856 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c4e77a-6bdd-41a4-8a57-b7eb6af064f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:53.825130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c4e77a-6bdd-41a4-8a57-b7eb6af064f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:53.825396 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c4e77a-6bdd-41a4-8a57-b7eb6af064f5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:53.826009 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c4e77a-6bdd-41a4-8a57-b7eb6af064f5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:53.828045 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c4e77a-6bdd-41a4-8a57-b7eb6af064f5 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:53.846040 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-1488873e-eaa1-481e-9d8a-7a86759e91b6 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:53.847310 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-1488873e-eaa1-481e-9d8a-7a86759e91b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:53.856406 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1212/2404] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:08:53.861479 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1193/2405] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:53 2022] POST /identity/v3/projects => generated 409 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:53.861662 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1488873e-eaa1-481e-9d8a-7a86759e91b6 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:53.862445 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c4e77a-6bdd-41a4-8a57-b7eb6af064f5 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:53.870809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1488873e-eaa1-481e-9d8a-7a86759e91b6 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:53.879218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:53.924292 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c4e77a-6bdd-41a4-8a57-b7eb6af064f5 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:53.929404 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ff7654-fbb2-436a-a516-5937229d0de0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:53.929910 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4a7985-2023-4174-8e6e-f4a2b8d8be38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:53.930038 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ff7654-fbb2-436a-a516-5937229d0de0 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:53.930271 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ff7654-fbb2-436a-a516-5937229d0de0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:53.930515 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4a7985-2023-4174-8e6e-f4a2b8d8be38 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:53.930643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ff7654-fbb2-436a-a516-5937229d0de0 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:53.930786 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4a7985-2023-4174-8e6e-f4a2b8d8be38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:53.931046 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4a7985-2023-4174-8e6e-f4a2b8d8be38 admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:53.931153 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ff7654-fbb2-436a-a516-5937229d0de0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:53.931677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4a7985-2023-4174-8e6e-f4a2b8d8be38 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:08:53.931999 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4a7985-2023-4174-8e6e-f4a2b8d8be38 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:53.933137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ff7654-fbb2-436a-a516-5937229d0de0 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:53.933569 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4a7985-2023-4174-8e6e-f4a2b8d8be38 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:53.945700 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1213/2406] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:08:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 30 01:08:53.952175 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4a7985-2023-4174-8e6e-f4a2b8d8be38 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:53.960195 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1194/2407] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:53 2022] POST /identity/v3/users => generated 305 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:08:53.963107 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4a7985-2023-4174-8e6e-f4a2b8d8be38 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:53.964030 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ff7654-fbb2-436a-a516-5937229d0de0 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:53.967825 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a350612-a0df-4c23-ba04-effab0570dc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:53.968550 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a350612-a0df-4c23-ba04-effab0570dc8 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:53.969131 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a350612-a0df-4c23-ba04-effab0570dc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:53.969504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a350612-a0df-4c23-ba04-effab0570dc8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:53.969990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:53.970178 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a350612-a0df-4c23-ba04-effab0570dc8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:53.972162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a350612-a0df-4c23-ba04-effab0570dc8 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:53.974046 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ff7654-fbb2-436a-a516-5937229d0de0 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:53.980363 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240d45db-77f7-409a-a4e3-91d0f9182cdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:53.981068 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240d45db-77f7-409a-a4e3-91d0f9182cdd admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:53.981340 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240d45db-77f7-409a-a4e3-91d0f9182cdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:53.981609 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240d45db-77f7-409a-a4e3-91d0f9182cdd admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:53.982237 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240d45db-77f7-409a-a4e3-91d0f9182cdd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:53.984381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240d45db-77f7-409a-a4e3-91d0f9182cdd admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:53.996148 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1195/2408] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:53 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:54.000286 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240d45db-77f7-409a-a4e3-91d0f9182cdd admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:54.002336 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1214/2409] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:53 2022] POST /identity/v3/projects => generated 365 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:08:54.006542 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a350612-a0df-4c23-ba04-effab0570dc8 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:54.010012 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240d45db-77f7-409a-a4e3-91d0f9182cdd admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:54.065624 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f1c7fa-5b91-47bf-82fd-66b06405422e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:54.066234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f1c7fa-5b91-47bf-82fd-66b06405422e admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.066504 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f1c7fa-5b91-47bf-82fd-66b06405422e admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.066822 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f1c7fa-5b91-47bf-82fd-66b06405422e admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.067453 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f1c7fa-5b91-47bf-82fd-66b06405422e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:54.069091 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f1c7fa-5b91-47bf-82fd-66b06405422e admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:54.070449 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a350612-a0df-4c23-ba04-effab0570dc8 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:54.075624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bf91ce-5409-4dd5-8af7-300388b42176 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:54.076330 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bf91ce-5409-4dd5-8af7-300388b42176 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.076598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bf91ce-5409-4dd5-8af7-300388b42176 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.076971 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bf91ce-5409-4dd5-8af7-300388b42176 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.077681 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bf91ce-5409-4dd5-8af7-300388b42176 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:54.079586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bf91ce-5409-4dd5-8af7-300388b42176 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:54.080224 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1196/2410] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:53 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:08:54.084214 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f1c7fa-5b91-47bf-82fd-66b06405422e admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:54.095122 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f1c7fa-5b91-47bf-82fd-66b06405422e admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:54.100314 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cb7990-e6b8-4410-949c-59d0ec0b1831 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:54.100879 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cb7990-e6b8-4410-949c-59d0ec0b1831 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.101157 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cb7990-e6b8-4410-949c-59d0ec0b1831 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.101429 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97cb7990-e6b8-4410-949c-59d0ec0b1831 admin admin] PATH_INFO: `/v3/projects/510a38bbcf8f47b1838036f68160b1a9/users/2406a55c5be54ed1992cb45616ef2e76/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.106503 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1215/2411] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:54 2022] POST /identity/v3/users => generated 283 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:08:54.107633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bf91ce-5409-4dd5-8af7-300388b42176 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:54.117054 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97cb7990-e6b8-4410-949c-59d0ec0b1831 admin admin] RBAC: Authorizing `identity:create_grant(project_id=510a38bbcf8f47b1838036f68160b1a9, user_id=2406a55c5be54ed1992cb45616ef2e76, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:54.117951 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bf91ce-5409-4dd5-8af7-300388b42176 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:54.119697 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97cb7990-e6b8-4410-949c-59d0ec0b1831 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:54.122643 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99cb95ea-a261-4483-9141-84c044155cfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:54.123228 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cb95ea-a261-4483-9141-84c044155cfc admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.123503 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cb95ea-a261-4483-9141-84c044155cfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.123787 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cb95ea-a261-4483-9141-84c044155cfc admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.124639 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99cb95ea-a261-4483-9141-84c044155cfc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:54.126453 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99cb95ea-a261-4483-9141-84c044155cfc admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:54.133943 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1197/2412] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:54 2022] PUT /identity/v3/projects/510a38bbcf8f47b1838036f68160b1a9/users/2406a55c5be54ed1992cb45616ef2e76/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:08:54.136947 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1216/2413] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:54 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:54.141018 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cb7990-e6b8-4410-949c-59d0ec0b1831 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:54.144085 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1864449-2188-4765-8059-63cf829ec0b2 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.144366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1864449-2188-4765-8059-63cf829ec0b2 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.144639 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1864449-2188-4765-8059-63cf829ec0b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.172876 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-e1864449-2188-4765-8059-63cf829ec0b2 None None] MFA Rules not processed for user `2406a55c5be54ed1992cb45616ef2e76`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:54.174089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e1864449-2188-4765-8059-63cf829ec0b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:54.208141 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97cb7990-e6b8-4410-949c-59d0ec0b1831 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:54.214434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731ceb48-e146-44ef-bbcd-a7a516d4b8f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:54.215130 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731ceb48-e146-44ef-bbcd-a7a516d4b8f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.215475 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731ceb48-e146-44ef-bbcd-a7a516d4b8f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.215909 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731ceb48-e146-44ef-bbcd-a7a516d4b8f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.216777 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731ceb48-e146-44ef-bbcd-a7a516d4b8f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:54.219035 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731ceb48-e146-44ef-bbcd-a7a516d4b8f4 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:54.229486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1198/2414] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:54 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:54.233855 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:54.234226 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731ceb48-e146-44ef-bbcd-a7a516d4b8f4 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:54.246457 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731ceb48-e146-44ef-bbcd-a7a516d4b8f4 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:54.251536 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08efbc2d-d292-46ba-8fa6-25cad966e820 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:54.252234 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08efbc2d-d292-46ba-8fa6-25cad966e820 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.252505 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08efbc2d-d292-46ba-8fa6-25cad966e820 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.252619 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1217/2415] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:54 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:08:54.252820 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08efbc2d-d292-46ba-8fa6-25cad966e820 admin admin] PATH_INFO: `/v3/projects/4d741226dc5b400189fda6f43d636697/users/296fc4cec81e428e90a49bbd363f998a/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.257237 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1864449-2188-4765-8059-63cf829ec0b2 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:54.267494 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1864449-2188-4765-8059-63cf829ec0b2 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:54.269250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08efbc2d-d292-46ba-8fa6-25cad966e820 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4d741226dc5b400189fda6f43d636697, user_id=296fc4cec81e428e90a49bbd363f998a, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:54.271935 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08efbc2d-d292-46ba-8fa6-25cad966e820 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:54.272083 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b81ac65-0038-45c1-adf1-3369d6fb22d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:54.272638 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b81ac65-0038-45c1-adf1-3369d6fb22d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.272911 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b81ac65-0038-45c1-adf1-3369d6fb22d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.273174 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b81ac65-0038-45c1-adf1-3369d6fb22d5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.273791 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b81ac65-0038-45c1-adf1-3369d6fb22d5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:54.275695 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b81ac65-0038-45c1-adf1-3369d6fb22d5 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:54.286163 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1199/2416] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:54 2022] PUT /identity/v3/projects/4d741226dc5b400189fda6f43d636697/users/296fc4cec81e428e90a49bbd363f998a/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:08:54.296918 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e46bb7-5910-44f3-b19a-8525d9003d90 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.297143 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e46bb7-5910-44f3-b19a-8525d9003d90 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.297499 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e46bb7-5910-44f3-b19a-8525d9003d90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.309432 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1218/2417] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:54 2022] POST /identity/v3/projects => generated 407 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:08:54.310440 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b81ac65-0038-45c1-adf1-3369d6fb22d5 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:54.325716 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-e5e46bb7-5910-44f3-b19a-8525d9003d90 None None] MFA Rules not processed for user `296fc4cec81e428e90a49bbd363f998a`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:54.326878 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-e5e46bb7-5910-44f3-b19a-8525d9003d90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:54.357462 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:54.379699 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b81ac65-0038-45c1-adf1-3369d6fb22d5 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:54.384542 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1fb9dd-dd1b-4459-ba8c-439df038fb50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:54.385150 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1fb9dd-dd1b-4459-ba8c-439df038fb50 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.385422 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1fb9dd-dd1b-4459-ba8c-439df038fb50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.385686 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1fb9dd-dd1b-4459-ba8c-439df038fb50 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.386312 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1fb9dd-dd1b-4459-ba8c-439df038fb50 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:54.388262 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1fb9dd-dd1b-4459-ba8c-439df038fb50 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:54.400371 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1200/2418] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:54 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:08:54.405107 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e46bb7-5910-44f3-b19a-8525d9003d90 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:54.411160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1219/2419] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:54 2022] POST /identity/v3/users => generated 310 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:54.414894 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1fb9dd-dd1b-4459-ba8c-439df038fb50 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:54.415825 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e46bb7-5910-44f3-b19a-8525d9003d90 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:54.420910 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f400c5e-93f7-4cde-aaa3-640dec39c306 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:54.421526 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f400c5e-93f7-4cde-aaa3-640dec39c306 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.421793 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f400c5e-93f7-4cde-aaa3-640dec39c306 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.422062 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f400c5e-93f7-4cde-aaa3-640dec39c306 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.422682 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f400c5e-93f7-4cde-aaa3-640dec39c306 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:54.424605 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f400c5e-93f7-4cde-aaa3-640dec39c306 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:54.424745 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1fb9dd-dd1b-4459-ba8c-439df038fb50 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:54.433770 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0584f26-cbd3-481f-9d2a-ead8cdec3a05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:54.435205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0584f26-cbd3-481f-9d2a-ead8cdec3a05 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.435902 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0584f26-cbd3-481f-9d2a-ead8cdec3a05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.436220 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0584f26-cbd3-481f-9d2a-ead8cdec3a05 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.437072 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0584f26-cbd3-481f-9d2a-ead8cdec3a05 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:54.439168 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0584f26-cbd3-481f-9d2a-ead8cdec3a05 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:54.450372 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1220/2420] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:54 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:08:54.453264 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1201/2421] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:54 2022] POST /identity/v3/projects => generated 361 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:08:54.454711 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0584f26-cbd3-481f-9d2a-ead8cdec3a05 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:54.457221 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f400c5e-93f7-4cde-aaa3-640dec39c306 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:54.464783 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0584f26-cbd3-481f-9d2a-ead8cdec3a05 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:54.518803 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f400c5e-93f7-4cde-aaa3-640dec39c306 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:54.520714 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c9adda-3a56-4e54-a680-52c07c2e542a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:54.521439 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c9adda-3a56-4e54-a680-52c07c2e542a admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.521716 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c9adda-3a56-4e54-a680-52c07c2e542a admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.521989 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c9adda-3a56-4e54-a680-52c07c2e542a admin admin] PATH_INFO: `/v3/projects/dc0d2c0478fe40abb1d87c868e0f2715/users/21d0d385bd8a4fa68b98081dc570120d/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.523591 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f58f489-ad6e-4430-807d-d9b5acb70db6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:54.524218 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f58f489-ad6e-4430-807d-d9b5acb70db6 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.524486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f58f489-ad6e-4430-807d-d9b5acb70db6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.524757 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f58f489-ad6e-4430-807d-d9b5acb70db6 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.525382 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f58f489-ad6e-4430-807d-d9b5acb70db6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:54.527299 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f58f489-ad6e-4430-807d-d9b5acb70db6 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:54.539028 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c9adda-3a56-4e54-a680-52c07c2e542a admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc0d2c0478fe40abb1d87c868e0f2715, user_id=21d0d385bd8a4fa68b98081dc570120d, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:54.542429 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c9adda-3a56-4e54-a680-52c07c2e542a admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:54.552680 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1202/2422] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:54 2022] POST /identity/v3/users => generated 287 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:54.554070 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f58f489-ad6e-4430-807d-d9b5acb70db6 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:54.556555 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1221/2423] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:54 2022] PUT /identity/v3/projects/dc0d2c0478fe40abb1d87c868e0f2715/users/21d0d385bd8a4fa68b98081dc570120d/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:54.560296 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c9adda-3a56-4e54-a680-52c07c2e542a admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:54.568192 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f58f489-ad6e-4430-807d-d9b5acb70db6 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:54.594293 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:54.625835 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c169845-db71-40ac-aa41-6faf5f21979a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:54.626179 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c9adda-3a56-4e54-a680-52c07c2e542a admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:54.626331 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c169845-db71-40ac-aa41-6faf5f21979a admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.626622 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c169845-db71-40ac-aa41-6faf5f21979a admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.626885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c169845-db71-40ac-aa41-6faf5f21979a admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.627529 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c169845-db71-40ac-aa41-6faf5f21979a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:54.629168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c169845-db71-40ac-aa41-6faf5f21979a admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:54.631107 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53abd4f5-e7d7-4003-b3e8-d9ded542dc1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:54.631710 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53abd4f5-e7d7-4003-b3e8-d9ded542dc1d admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.632008 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53abd4f5-e7d7-4003-b3e8-d9ded542dc1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.632275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53abd4f5-e7d7-4003-b3e8-d9ded542dc1d admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.633048 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53abd4f5-e7d7-4003-b3e8-d9ded542dc1d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:54.634609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53abd4f5-e7d7-4003-b3e8-d9ded542dc1d admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:54.639347 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1203/2424] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:54 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:08:54.644041 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c169845-db71-40ac-aa41-6faf5f21979a admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:54.645307 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1222/2425] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:54 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:08:54.649160 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53abd4f5-e7d7-4003-b3e8-d9ded542dc1d admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:54.654248 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c169845-db71-40ac-aa41-6faf5f21979a admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:54.659133 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53abd4f5-e7d7-4003-b3e8-d9ded542dc1d admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:54.659405 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94a88fd-982f-486b-87f9-c87691524e0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:54.660095 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94a88fd-982f-486b-87f9-c87691524e0a admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.660448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94a88fd-982f-486b-87f9-c87691524e0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.660681 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94a88fd-982f-486b-87f9-c87691524e0a admin admin] PATH_INFO: `/v3/projects/ba6013ffae7649a69eede9428adcc3b0/users/e1741495f4b54ad88582ec28016784b8/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.664496 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16bb413-439a-4a6a-93fe-a34b2f296c94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:54.665182 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16bb413-439a-4a6a-93fe-a34b2f296c94 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.665456 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16bb413-439a-4a6a-93fe-a34b2f296c94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.665723 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16bb413-439a-4a6a-93fe-a34b2f296c94 admin admin] PATH_INFO: `/v3/projects/dc0d2c0478fe40abb1d87c868e0f2715/users/21d0d385bd8a4fa68b98081dc570120d/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.667684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16bb413-439a-4a6a-93fe-a34b2f296c94 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc0d2c0478fe40abb1d87c868e0f2715, user_id=21d0d385bd8a4fa68b98081dc570120d, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:54.670483 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16bb413-439a-4a6a-93fe-a34b2f296c94 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:54.676391 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94a88fd-982f-486b-87f9-c87691524e0a admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba6013ffae7649a69eede9428adcc3b0, user_id=e1741495f4b54ad88582ec28016784b8, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:54.679064 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94a88fd-982f-486b-87f9-c87691524e0a admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:54.681704 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1223/2426] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:54 2022] PUT /identity/v3/projects/dc0d2c0478fe40abb1d87c868e0f2715/users/21d0d385bd8a4fa68b98081dc570120d/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:08:54.693472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6472222-f332-4267-9bf0-4c6e30aec678 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.693768 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6472222-f332-4267-9bf0-4c6e30aec678 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.694033 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6472222-f332-4267-9bf0-4c6e30aec678 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.694862 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1204/2427] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:54 2022] PUT /identity/v3/projects/ba6013ffae7649a69eede9428adcc3b0/users/e1741495f4b54ad88582ec28016784b8/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:08:54.698855 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94a88fd-982f-486b-87f9-c87691524e0a admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:54.707667 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:54.723440 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-f6472222-f332-4267-9bf0-4c6e30aec678 None None] MFA Rules not processed for user `21d0d385bd8a4fa68b98081dc570120d`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:54.724741 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-f6472222-f332-4267-9bf0-4c6e30aec678 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:54.764795 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94a88fd-982f-486b-87f9-c87691524e0a admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:54.770439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e445586-4faa-47a6-9ad6-af88aa8b3243 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:54.771041 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e445586-4faa-47a6-9ad6-af88aa8b3243 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.771300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e445586-4faa-47a6-9ad6-af88aa8b3243 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.771764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e445586-4faa-47a6-9ad6-af88aa8b3243 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.772923 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e445586-4faa-47a6-9ad6-af88aa8b3243 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:54.774528 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e445586-4faa-47a6-9ad6-af88aa8b3243 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:54.785188 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1205/2428] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:54 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:54.789269 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e445586-4faa-47a6-9ad6-af88aa8b3243 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:54.796074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1224/2429] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:54 2022] POST /identity/v3/auth/tokens => generated 3351 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:08:54.799024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e445586-4faa-47a6-9ad6-af88aa8b3243 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:54.803755 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7166ce46-d3c5-495a-9d6e-3111f7a12ae2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:54.804396 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7166ce46-d3c5-495a-9d6e-3111f7a12ae2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.804666 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7166ce46-d3c5-495a-9d6e-3111f7a12ae2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.804932 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7166ce46-d3c5-495a-9d6e-3111f7a12ae2 admin admin] PATH_INFO: `/v3/projects/ba6013ffae7649a69eede9428adcc3b0/users/e1741495f4b54ad88582ec28016784b8/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.806797 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7166ce46-d3c5-495a-9d6e-3111f7a12ae2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba6013ffae7649a69eede9428adcc3b0, user_id=e1741495f4b54ad88582ec28016784b8, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:54.807112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6472222-f332-4267-9bf0-4c6e30aec678 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:54.808565 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-f6472222-f332-4267-9bf0-4c6e30aec678 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:54.809437 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7166ce46-d3c5-495a-9d6e-3111f7a12ae2 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:54.820948 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1206/2430] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:54 2022] PUT /identity/v3/projects/ba6013ffae7649a69eede9428adcc3b0/users/e1741495f4b54ad88582ec28016784b8/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:54.832517 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad057e7-f2fc-45d4-a6f6-6143be07b4af None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.833755 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad057e7-f2fc-45d4-a6f6-6143be07b4af None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.834035 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad057e7-f2fc-45d4-a6f6-6143be07b4af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.862732 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-8ad057e7-f2fc-45d4-a6f6-6143be07b4af None None] MFA Rules not processed for user `e1741495f4b54ad88582ec28016784b8`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:54.863995 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-8ad057e7-f2fc-45d4-a6f6-6143be07b4af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:54.866198 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6472222-f332-4267-9bf0-4c6e30aec678 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:54.913847 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b8bb54-f636-4966-8051-4667ed636f94 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:54.914451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b8bb54-f636-4966-8051-4667ed636f94 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.914783 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b8bb54-f636-4966-8051-4667ed636f94 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.915112 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b8bb54-f636-4966-8051-4667ed636f94 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.923308 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:54.932519 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1207/2431] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:54 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:54.944535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad057e7-f2fc-45d4-a6f6-6143be07b4af None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:54.946099 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-8ad057e7-f2fc-45d4-a6f6-6143be07b4af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:54.958406 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad057e7-f2fc-45d4-a6f6-6143be07b4af None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:54.962472 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe325e4-467d-4b9b-986e-c690926d9dc0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:54.963071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe325e4-467d-4b9b-986e-c690926d9dc0 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:54.963345 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe325e4-467d-4b9b-986e-c690926d9dc0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:54.963613 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe325e4-467d-4b9b-986e-c690926d9dc0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:54.972959 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe325e4-467d-4b9b-986e-c690926d9dc0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:54.974384 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe325e4-467d-4b9b-986e-c690926d9dc0 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:54.978170 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b8bb54-f636-4966-8051-4667ed636f94 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:54.979606 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b8bb54-f636-4966-8051-4667ed636f94 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:54.983774 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1208/2432] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:54 2022] GET /identity/v3/auth/tokens => generated 3305 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:54.989181 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1225/2433] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:54 2022] GET /identity/v3/auth/tokens => generated 3351 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 01:08:55.066579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe325e4-467d-4b9b-986e-c690926d9dc0 service cinder] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:55.075593 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe325e4-467d-4b9b-986e-c690926d9dc0 service cinder] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:55.079663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13b4bf2-674e-4857-8d3c-5cda2e82bcb9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:55.080674 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13b4bf2-674e-4857-8d3c-5cda2e82bcb9 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:55.080956 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13b4bf2-674e-4857-8d3c-5cda2e82bcb9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:55.081220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13b4bf2-674e-4857-8d3c-5cda2e82bcb9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:55.143933 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13b4bf2-674e-4857-8d3c-5cda2e82bcb9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:55.143933 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13b4bf2-674e-4857-8d3c-5cda2e82bcb9 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:55.145651 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:55.151540 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1209/2434] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:55 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:55.151946 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b8bb54-f636-4966-8051-4667ed636f94 service cinder] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:55.161495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b8bb54-f636-4966-8051-4667ed636f94 service cinder] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:55.165451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6523ec8-374e-4613-b209-1dad1a6be98d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:55.165977 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6523ec8-374e-4613-b209-1dad1a6be98d service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:55.166247 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6523ec8-374e-4613-b209-1dad1a6be98d service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:55.166509 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6523ec8-374e-4613-b209-1dad1a6be98d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:55.226535 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6523ec8-374e-4613-b209-1dad1a6be98d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:55.228003 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6523ec8-374e-4613-b209-1dad1a6be98d service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:55.237149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1226/2435] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:55 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:57.950653 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13b4bf2-674e-4857-8d3c-5cda2e82bcb9 service cinder] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:58.017369 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13b4bf2-674e-4857-8d3c-5cda2e82bcb9 service cinder] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:58.022664 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77677c74-eef2-4865-9381-cc35ddd8d11c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:58.023290 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77677c74-eef2-4865-9381-cc35ddd8d11c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:58.023559 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77677c74-eef2-4865-9381-cc35ddd8d11c admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:58.024006 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77677c74-eef2-4865-9381-cc35ddd8d11c admin admin] PATH_INFO: `/v3/users/296fc4cec81e428e90a49bbd363f998a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:58.025388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77677c74-eef2-4865-9381-cc35ddd8d11c admin admin] RBAC: Authorizing `identity:delete_user(user_id=296fc4cec81e428e90a49bbd363f998a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:58.027331 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77677c74-eef2-4865-9381-cc35ddd8d11c admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:58.080547 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-77677c74-eef2-4865-9381-cc35ddd8d11c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '296fc4cec81e428e90a49bbd363f998a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:58.090202 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-77677c74-eef2-4865-9381-cc35ddd8d11c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '296fc4cec81e428e90a49bbd363f998a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:58.090810 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-77677c74-eef2-4865-9381-cc35ddd8d11c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '296fc4cec81e428e90a49bbd363f998a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:58.098263 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-77677c74-eef2-4865-9381-cc35ddd8d11c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '296fc4cec81e428e90a49bbd363f998a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:58.109732 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-77677c74-eef2-4865-9381-cc35ddd8d11c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '296fc4cec81e428e90a49bbd363f998a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:58.115565 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1210/2436] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:57 2022] DELETE /identity/v3/users/296fc4cec81e428e90a49bbd363f998a => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:58.120400 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04343a9-0dfc-4ad8-b592-96899ad1a038 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:58.120884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04343a9-0dfc-4ad8-b592-96899ad1a038 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:58.121007 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04343a9-0dfc-4ad8-b592-96899ad1a038 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:58.140668 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-f04343a9-0dfc-4ad8-b592-96899ad1a038 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:58.141993 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-f04343a9-0dfc-4ad8-b592-96899ad1a038 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:58.203133 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1227/2437] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:58.213842 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77677c74-eef2-4865-9381-cc35ddd8d11c admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:58.215360 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-77677c74-eef2-4865-9381-cc35ddd8d11c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:58.276083 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77677c74-eef2-4865-9381-cc35ddd8d11c admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:58.280151 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc64534-0cea-471e-9b08-9ac804613036 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:58.280751 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc64534-0cea-471e-9b08-9ac804613036 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:58.281498 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc64534-0cea-471e-9b08-9ac804613036 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:58.281764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc64534-0cea-471e-9b08-9ac804613036 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:58.292920 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc64534-0cea-471e-9b08-9ac804613036 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:58.295303 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc64534-0cea-471e-9b08-9ac804613036 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:58.305899 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1211/2438] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:08:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:08:58.595807 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04343a9-0dfc-4ad8-b592-96899ad1a038 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:58.597313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-f04343a9-0dfc-4ad8-b592-96899ad1a038 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:58.609485 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04343a9-0dfc-4ad8-b592-96899ad1a038 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:58.614649 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8e3c1f-328e-4cba-909c-cd4cea45ec2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:58.615356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8e3c1f-328e-4cba-909c-cd4cea45ec2d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:58.615688 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8e3c1f-328e-4cba-909c-cd4cea45ec2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:58.616046 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8e3c1f-328e-4cba-909c-cd4cea45ec2d admin admin] PATH_INFO: `/v3/projects/4d741226dc5b400189fda6f43d636697` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:58.617329 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8e3c1f-328e-4cba-909c-cd4cea45ec2d admin admin] RBAC: Authorizing `identity:delete_project(project_id=4d741226dc5b400189fda6f43d636697)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:58.620598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8e3c1f-328e-4cba-909c-cd4cea45ec2d admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:58.667152 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:58.675043 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ae8e3c1f-328e-4cba-909c-cd4cea45ec2d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4d741226dc5b400189fda6f43d636697', 'request_id': 'req-ae8e3c1f-328e-4cba-909c-cd4cea45ec2d'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:58.684387 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ae8e3c1f-328e-4cba-909c-cd4cea45ec2d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4d741226dc5b400189fda6f43d636697', 'request_id': 'req-ae8e3c1f-328e-4cba-909c-cd4cea45ec2d'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:58.690602 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ae8e3c1f-328e-4cba-909c-cd4cea45ec2d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4d741226dc5b400189fda6f43d636697', 'request_id': 'req-ae8e3c1f-328e-4cba-909c-cd4cea45ec2d'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:58.702592 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ae8e3c1f-328e-4cba-909c-cd4cea45ec2d admin admin] The token cache is being invalidate because project 4d741226dc5b400189fda6f43d636697 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:58.702873 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ae8e3c1f-328e-4cba-909c-cd4cea45ec2d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:58.704020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-ae8e3c1f-328e-4cba-909c-cd4cea45ec2d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:58.705548 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1228/2439] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:58 2022] DELETE /identity/v3/projects/4d741226dc5b400189fda6f43d636697 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:58.710206 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc64534-0cea-471e-9b08-9ac804613036 service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:58.721861 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc64534-0cea-471e-9b08-9ac804613036 service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:58.742876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:58.798621 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fc8ed1-addc-4e37-b551-96da76b5c3e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:58.799260 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fc8ed1-addc-4e37-b551-96da76b5c3e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:58.799539 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fc8ed1-addc-4e37-b551-96da76b5c3e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:58.799852 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fc8ed1-addc-4e37-b551-96da76b5c3e8 admin admin] PATH_INFO: `/v3/users/e1741495f4b54ad88582ec28016784b8` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:58.800977 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fc8ed1-addc-4e37-b551-96da76b5c3e8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e1741495f4b54ad88582ec28016784b8)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:58.802926 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fc8ed1-addc-4e37-b551-96da76b5c3e8 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:58.860160 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-56fc8ed1-addc-4e37-b551-96da76b5c3e8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e1741495f4b54ad88582ec28016784b8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:58.869929 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-56fc8ed1-addc-4e37-b551-96da76b5c3e8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e1741495f4b54ad88582ec28016784b8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:58.870583 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-56fc8ed1-addc-4e37-b551-96da76b5c3e8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e1741495f4b54ad88582ec28016784b8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:58.878253 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-56fc8ed1-addc-4e37-b551-96da76b5c3e8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e1741495f4b54ad88582ec28016784b8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:58.892282 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-56fc8ed1-addc-4e37-b551-96da76b5c3e8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e1741495f4b54ad88582ec28016784b8'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:58.898357 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1212/2440] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:08:58 2022] DELETE /identity/v3/users/e1741495f4b54ad88582ec28016784b8 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:59.180794 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8e3c1f-328e-4cba-909c-cd4cea45ec2d admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:59.182244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-ae8e3c1f-328e-4cba-909c-cd4cea45ec2d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:59.198593 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8e3c1f-328e-4cba-909c-cd4cea45ec2d admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:59.259022 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148925e3-e438-4c6f-825e-6595cfb18a8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:59.259795 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148925e3-e438-4c6f-825e-6595cfb18a8e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:59.260071 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148925e3-e438-4c6f-825e-6595cfb18a8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:59.260337 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148925e3-e438-4c6f-825e-6595cfb18a8e admin admin] PATH_INFO: `/v3/projects/ba6013ffae7649a69eede9428adcc3b0` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:59.261440 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148925e3-e438-4c6f-825e-6595cfb18a8e admin admin] RBAC: Authorizing `identity:delete_project(project_id=ba6013ffae7649a69eede9428adcc3b0)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:59.263334 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148925e3-e438-4c6f-825e-6595cfb18a8e admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:59.312374 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-148925e3-e438-4c6f-825e-6595cfb18a8e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ba6013ffae7649a69eede9428adcc3b0', 'request_id': 'req-148925e3-e438-4c6f-825e-6595cfb18a8e'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:59.321327 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-148925e3-e438-4c6f-825e-6595cfb18a8e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ba6013ffae7649a69eede9428adcc3b0', 'request_id': 'req-148925e3-e438-4c6f-825e-6595cfb18a8e'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:59.327659 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-148925e3-e438-4c6f-825e-6595cfb18a8e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ba6013ffae7649a69eede9428adcc3b0', 'request_id': 'req-148925e3-e438-4c6f-825e-6595cfb18a8e'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:59.341430 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-148925e3-e438-4c6f-825e-6595cfb18a8e admin admin] The token cache is being invalidate because project ba6013ffae7649a69eede9428adcc3b0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:08:59.341811 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-148925e3-e438-4c6f-825e-6595cfb18a8e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:59.342486 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-148925e3-e438-4c6f-825e-6595cfb18a8e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:08:59.344350 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1229/2441] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:08:59 2022] DELETE /identity/v3/projects/ba6013ffae7649a69eede9428adcc3b0 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:59.349497 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8fa51f-28fa-407e-87f9-59ec0896711f None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:59.349497 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8fa51f-28fa-407e-87f9-59ec0896711f None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:59.349971 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8fa51f-28fa-407e-87f9-59ec0896711f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:59.369934 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-7c8fa51f-28fa-407e-87f9-59ec0896711f None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:59.371136 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-7c8fa51f-28fa-407e-87f9-59ec0896711f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:59.432515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1213/2442] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:59.440939 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92901412-1629-45ce-b9b1-3609ed0666c5 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:59.441219 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92901412-1629-45ce-b9b1-3609ed0666c5 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:59.441488 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92901412-1629-45ce-b9b1-3609ed0666c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:59.461941 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-92901412-1629-45ce-b9b1-3609ed0666c5 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:59.463130 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-92901412-1629-45ce-b9b1-3609ed0666c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:59.471733 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1230/2443] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:59.476563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8fa51f-28fa-407e-87f9-59ec0896711f None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:59.491495 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8fa51f-28fa-407e-87f9-59ec0896711f None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:59.499871 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd974142-1b2c-433b-a21d-3f232c5cc6bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:59.500507 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd974142-1b2c-433b-a21d-3f232c5cc6bd admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:59.500782 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd974142-1b2c-433b-a21d-3f232c5cc6bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:59.501045 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd974142-1b2c-433b-a21d-3f232c5cc6bd admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:59.501687 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd974142-1b2c-433b-a21d-3f232c5cc6bd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:08:59.501983 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd974142-1b2c-433b-a21d-3f232c5cc6bd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:59.503567 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd974142-1b2c-433b-a21d-3f232c5cc6bd admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:59.516538 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1214/2444] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:08:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 01:08:59.521682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92901412-1629-45ce-b9b1-3609ed0666c5 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:59.533830 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92901412-1629-45ce-b9b1-3609ed0666c5 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:59.538988 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444b7be8-3029-49f9-b5d8-cdf4a7cad8ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:59.539580 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444b7be8-3029-49f9-b5d8-cdf4a7cad8ac admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:59.539892 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444b7be8-3029-49f9-b5d8-cdf4a7cad8ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:59.540148 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444b7be8-3029-49f9-b5d8-cdf4a7cad8ac admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:59.540767 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444b7be8-3029-49f9-b5d8-cdf4a7cad8ac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:59.542635 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444b7be8-3029-49f9-b5d8-cdf4a7cad8ac admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:59.571832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1231/2445] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:59 2022] POST /identity/v3/projects => generated 385 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:59.576362 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd974142-1b2c-433b-a21d-3f232c5cc6bd admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:59.587737 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd974142-1b2c-433b-a21d-3f232c5cc6bd admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:59.613323 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:59.646396 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8685aad2-241c-4c2c-bbe2-307d35c3ccc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:59.647016 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8685aad2-241c-4c2c-bbe2-307d35c3ccc5 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:59.647355 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8685aad2-241c-4c2c-bbe2-307d35c3ccc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:59.647617 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8685aad2-241c-4c2c-bbe2-307d35c3ccc5 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:59.648288 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8685aad2-241c-4c2c-bbe2-307d35c3ccc5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:59.650256 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8685aad2-241c-4c2c-bbe2-307d35c3ccc5 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:59.677344 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1215/2446] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:08:59 2022] POST /identity/v3/users => generated 293 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:08:59.682952 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444b7be8-3029-49f9-b5d8-cdf4a7cad8ac admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:59.693902 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444b7be8-3029-49f9-b5d8-cdf4a7cad8ac admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:59.699568 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e574ec-2db7-4a37-8576-11b6f5ac8742 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:59.700222 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e574ec-2db7-4a37-8576-11b6f5ac8742 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:59.700504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e574ec-2db7-4a37-8576-11b6f5ac8742 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:59.700769 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e574ec-2db7-4a37-8576-11b6f5ac8742 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:59.701400 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e574ec-2db7-4a37-8576-11b6f5ac8742 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:59.703469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e574ec-2db7-4a37-8576-11b6f5ac8742 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:59.714277 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1232/2447] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:59 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:59.718480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8685aad2-241c-4c2c-bbe2-307d35c3ccc5 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:59.729300 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8685aad2-241c-4c2c-bbe2-307d35c3ccc5 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:59.735096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef695cba-1042-41a8-a11d-d52e63f8683b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:59.735702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef695cba-1042-41a8-a11d-d52e63f8683b admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:59.736033 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef695cba-1042-41a8-a11d-d52e63f8683b admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:59.736309 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef695cba-1042-41a8-a11d-d52e63f8683b admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:59.736922 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef695cba-1042-41a8-a11d-d52e63f8683b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:59.738516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef695cba-1042-41a8-a11d-d52e63f8683b admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:59.749321 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1216/2448] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:08:59 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:08:59.753703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e574ec-2db7-4a37-8576-11b6f5ac8742 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:08:59.763653 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e574ec-2db7-4a37-8576-11b6f5ac8742 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:08:59.768337 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e047ff-5d9f-4794-b6e0-ca58bfe265f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:08:59.768979 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e047ff-5d9f-4794-b6e0-ca58bfe265f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:59.769246 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e047ff-5d9f-4794-b6e0-ca58bfe265f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:59.769507 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e047ff-5d9f-4794-b6e0-ca58bfe265f7 admin admin] PATH_INFO: `/v3/projects/f7745fb95e6b4956a08b711aeec7832b/users/11e58f202d8a4249b2905961b1140b99/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:59.784204 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e047ff-5d9f-4794-b6e0-ca58bfe265f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7745fb95e6b4956a08b711aeec7832b, user_id=11e58f202d8a4249b2905961b1140b99, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:08:59.786855 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e047ff-5d9f-4794-b6e0-ca58bfe265f7 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:08:59.789313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:08:59.810489 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1233/2449] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:08:59 2022] PUT /identity/v3/projects/f7745fb95e6b4956a08b711aeec7832b/users/11e58f202d8a4249b2905961b1140b99/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:08:59.822819 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754dcf72-5552-4696-8fc0-408e11cd753e None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:08:59.823153 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754dcf72-5552-4696-8fc0-408e11cd753e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:08:59.823521 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754dcf72-5552-4696-8fc0-408e11cd753e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:08:59.860133 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-754dcf72-5552-4696-8fc0-408e11cd753e None None] MFA Rules not processed for user `11e58f202d8a4249b2905961b1140b99`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:08:59.861834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-754dcf72-5552-4696-8fc0-408e11cd753e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:08:59.969405 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1217/2450] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:08:59 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:08:59.975756 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e047ff-5d9f-4794-b6e0-ca58bfe265f7 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:00.054106 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e047ff-5d9f-4794-b6e0-ca58bfe265f7 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:00.059478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b193005-1199-4804-a0ee-a028db4db0e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:00.060271 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b193005-1199-4804-a0ee-a028db4db0e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:00.060545 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b193005-1199-4804-a0ee-a028db4db0e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:00.060806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b193005-1199-4804-a0ee-a028db4db0e4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:00.061449 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b193005-1199-4804-a0ee-a028db4db0e4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:00.063522 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b193005-1199-4804-a0ee-a028db4db0e4 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:00.103317 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1234/2451] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:08:59 2022] POST /identity/v3/projects => generated 387 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:09:00.107941 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754dcf72-5552-4696-8fc0-408e11cd753e None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:00.128116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754dcf72-5552-4696-8fc0-408e11cd753e None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:00.205620 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f850e771-ee08-4ec9-8073-2f980e2846bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:00.206298 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f850e771-ee08-4ec9-8073-2f980e2846bf admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:00.206601 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f850e771-ee08-4ec9-8073-2f980e2846bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:00.206931 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f850e771-ee08-4ec9-8073-2f980e2846bf admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:00.207769 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f850e771-ee08-4ec9-8073-2f980e2846bf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:00.210215 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f850e771-ee08-4ec9-8073-2f980e2846bf admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:00.236535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1218/2452] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:09:00 2022] POST /identity/v3/users => generated 294 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:09:00.240656 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b193005-1199-4804-a0ee-a028db4db0e4 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:00.251708 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b193005-1199-4804-a0ee-a028db4db0e4 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:00.256956 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcc3d09-66f1-42cd-8609-ac09e40f1381 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:00.257608 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcc3d09-66f1-42cd-8609-ac09e40f1381 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:00.257888 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcc3d09-66f1-42cd-8609-ac09e40f1381 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:00.258296 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcc3d09-66f1-42cd-8609-ac09e40f1381 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:00.258788 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcc3d09-66f1-42cd-8609-ac09e40f1381 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:00.260448 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcc3d09-66f1-42cd-8609-ac09e40f1381 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:00.272824 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1235/2453] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:00 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:09:00.277293 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f850e771-ee08-4ec9-8073-2f980e2846bf admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:00.288637 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f850e771-ee08-4ec9-8073-2f980e2846bf admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:00.296695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fd1a24-c5f7-490d-b293-b354eb79a96d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:00.298020 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fd1a24-c5f7-490d-b293-b354eb79a96d admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:00.298663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fd1a24-c5f7-490d-b293-b354eb79a96d admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:00.299137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fd1a24-c5f7-490d-b293-b354eb79a96d admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:00.300714 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fd1a24-c5f7-490d-b293-b354eb79a96d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:00.305145 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fd1a24-c5f7-490d-b293-b354eb79a96d admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:00.325486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1219/2454] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:00 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:09:00.330047 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcc3d09-66f1-42cd-8609-ac09e40f1381 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:00.345825 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcc3d09-66f1-42cd-8609-ac09e40f1381 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:00.349529 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68af834b-0e1c-4860-abde-b49dce497c4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:00.350226 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68af834b-0e1c-4860-abde-b49dce497c4d admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:00.350503 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68af834b-0e1c-4860-abde-b49dce497c4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:00.350923 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68af834b-0e1c-4860-abde-b49dce497c4d admin admin] PATH_INFO: `/v3/projects/7b347736df4b4907adc4e112f9c81aa6/users/802474ffacbb40d9b202d187a439fe55/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:00.368189 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68af834b-0e1c-4860-abde-b49dce497c4d admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b347736df4b4907adc4e112f9c81aa6, user_id=802474ffacbb40d9b202d187a439fe55, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:00.370898 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68af834b-0e1c-4860-abde-b49dce497c4d admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:00.388006 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1236/2455] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:09:00 2022] PUT /identity/v3/projects/7b347736df4b4907adc4e112f9c81aa6/users/802474ffacbb40d9b202d187a439fe55/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:00.400391 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8e7587-4c10-4606-a7d7-4933dc2ea8b9 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:00.400989 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8e7587-4c10-4606-a7d7-4933dc2ea8b9 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:00.400989 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8e7587-4c10-4606-a7d7-4933dc2ea8b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:00.432056 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-5d8e7587-4c10-4606-a7d7-4933dc2ea8b9 None None] MFA Rules not processed for user `802474ffacbb40d9b202d187a439fe55`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:00.433282 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-5d8e7587-4c10-4606-a7d7-4933dc2ea8b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:00.438022 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:00.506730 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1220/2456] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:00 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:00.511746 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68af834b-0e1c-4860-abde-b49dce497c4d admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:00.586659 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68af834b-0e1c-4860-abde-b49dce497c4d admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:00.594687 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39622290-dbf2-40dd-9f51-a3e4f1897d11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:00.595629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39622290-dbf2-40dd-9f51-a3e4f1897d11 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:00.595968 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39622290-dbf2-40dd-9f51-a3e4f1897d11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:00.596189 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39622290-dbf2-40dd-9f51-a3e4f1897d11 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:00.596842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39622290-dbf2-40dd-9f51-a3e4f1897d11 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:00.599446 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39622290-dbf2-40dd-9f51-a3e4f1897d11 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:00.639184 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1237/2457] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:09:00 2022] POST /identity/v3/projects => generated 385 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:09:00.644725 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8e7587-4c10-4606-a7d7-4933dc2ea8b9 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:00.656325 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8e7587-4c10-4606-a7d7-4933dc2ea8b9 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:00.716698 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe260d7-e8ac-44c5-875a-a9743db78224 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:00.717340 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe260d7-e8ac-44c5-875a-a9743db78224 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:00.717614 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe260d7-e8ac-44c5-875a-a9743db78224 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:00.717876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe260d7-e8ac-44c5-875a-a9743db78224 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:00.718540 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe260d7-e8ac-44c5-875a-a9743db78224 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:00.720518 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe260d7-e8ac-44c5-875a-a9743db78224 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:00.745984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1221/2458] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:09:00 2022] POST /identity/v3/users => generated 299 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:09:00.751764 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39622290-dbf2-40dd-9f51-a3e4f1897d11 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:00.765769 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39622290-dbf2-40dd-9f51-a3e4f1897d11 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:00.771851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e27760-0f9c-447f-965b-559d17f1a481 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:00.772478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e27760-0f9c-447f-965b-559d17f1a481 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:00.772766 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e27760-0f9c-447f-965b-559d17f1a481 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:00.772986 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e27760-0f9c-447f-965b-559d17f1a481 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:00.773603 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e27760-0f9c-447f-965b-559d17f1a481 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:00.775449 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e27760-0f9c-447f-965b-559d17f1a481 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:00.787353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1238/2459] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:00 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:09:00.792325 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe260d7-e8ac-44c5-875a-a9743db78224 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:00.807505 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe260d7-e8ac-44c5-875a-a9743db78224 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:00.814111 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3efbd08-b357-48e3-9aa3-5b1be98b1ff4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:00.814934 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3efbd08-b357-48e3-9aa3-5b1be98b1ff4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:00.815222 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3efbd08-b357-48e3-9aa3-5b1be98b1ff4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:00.816116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3efbd08-b357-48e3-9aa3-5b1be98b1ff4 admin admin] PATH_INFO: `/v3/projects/53cdbef41fdf42bd83b7ca286b22d97d/users/9817e4a34e3745c0bbd638968cadacc7/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:00.836527 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3efbd08-b357-48e3-9aa3-5b1be98b1ff4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=53cdbef41fdf42bd83b7ca286b22d97d, user_id=9817e4a34e3745c0bbd638968cadacc7, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:00.839184 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3efbd08-b357-48e3-9aa3-5b1be98b1ff4 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:00.857362 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1222/2460] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:09:00 2022] PUT /identity/v3/projects/53cdbef41fdf42bd83b7ca286b22d97d/users/9817e4a34e3745c0bbd638968cadacc7/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:00.861567 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e27760-0f9c-447f-965b-559d17f1a481 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:00.873403 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e27760-0f9c-447f-965b-559d17f1a481 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:00.887415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:00.933886 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c4ea07-9d5b-41fa-ac29-6081962f69bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:00.934520 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c4ea07-9d5b-41fa-ac29-6081962f69bb admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:00.934795 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c4ea07-9d5b-41fa-ac29-6081962f69bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:00.935076 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c4ea07-9d5b-41fa-ac29-6081962f69bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:00.935706 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c4ea07-9d5b-41fa-ac29-6081962f69bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:00.937475 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c4ea07-9d5b-41fa-ac29-6081962f69bb admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:00.950959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1239/2461] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:00 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:09:00.952119 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3efbd08-b357-48e3-9aa3-5b1be98b1ff4 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:00.963173 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3efbd08-b357-48e3-9aa3-5b1be98b1ff4 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:00.968422 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2494b64b-8b7c-4e17-9286-ac08b1be131a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:00.969047 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2494b64b-8b7c-4e17-9286-ac08b1be131a admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:00.969349 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2494b64b-8b7c-4e17-9286-ac08b1be131a admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:00.969621 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2494b64b-8b7c-4e17-9286-ac08b1be131a admin admin] PATH_INFO: `/v3/projects/53cdbef41fdf42bd83b7ca286b22d97d/users/9817e4a34e3745c0bbd638968cadacc7/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:00.971704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2494b64b-8b7c-4e17-9286-ac08b1be131a admin admin] RBAC: Authorizing `identity:create_grant(project_id=53cdbef41fdf42bd83b7ca286b22d97d, user_id=9817e4a34e3745c0bbd638968cadacc7, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:00.974366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2494b64b-8b7c-4e17-9286-ac08b1be131a admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:00.987047 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1223/2462] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:09:00 2022] PUT /identity/v3/projects/53cdbef41fdf42bd83b7ca286b22d97d/users/9817e4a34e3745c0bbd638968cadacc7/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:00.998004 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74f0725-a79a-4915-911d-a094a95d2ae1 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:00.998664 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74f0725-a79a-4915-911d-a094a95d2ae1 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:00.998945 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74f0725-a79a-4915-911d-a094a95d2ae1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:01.029820 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-d74f0725-a79a-4915-911d-a094a95d2ae1 None None] MFA Rules not processed for user `9817e4a34e3745c0bbd638968cadacc7`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:01.031338 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-d74f0725-a79a-4915-911d-a094a95d2ae1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:01.103449 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1240/2463] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:00 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:09:01.115755 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2494b64b-8b7c-4e17-9286-ac08b1be131a admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:01.117414 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-2494b64b-8b7c-4e17-9286-ac08b1be131a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:01.184928 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2494b64b-8b7c-4e17-9286-ac08b1be131a admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:01.189512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf261ea-efda-4229-b839-f8f14c605895 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:01.190095 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf261ea-efda-4229-b839-f8f14c605895 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:01.190873 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf261ea-efda-4229-b839-f8f14c605895 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:01.190873 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf261ea-efda-4229-b839-f8f14c605895 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:01.199902 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf261ea-efda-4229-b839-f8f14c605895 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:01.201477 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf261ea-efda-4229-b839-f8f14c605895 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:01.208329 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:01.217831 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1224/2464] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:09:01 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:09:01.772422 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74f0725-a79a-4915-911d-a094a95d2ae1 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:01.785137 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74f0725-a79a-4915-911d-a094a95d2ae1 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:01.789282 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416c9d17-1764-4a59-917e-699370246d85 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:01.789905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416c9d17-1764-4a59-917e-699370246d85 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:01.790178 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416c9d17-1764-4a59-917e-699370246d85 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:01.790443 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416c9d17-1764-4a59-917e-699370246d85 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:01.854580 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416c9d17-1764-4a59-917e-699370246d85 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:01.856120 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416c9d17-1764-4a59-917e-699370246d85 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:01.866129 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1241/2465] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:09:01 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:09:03.721261 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf261ea-efda-4229-b839-f8f14c605895 service cinder] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:03.722871 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-fdf261ea-efda-4229-b839-f8f14c605895 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:03.791488 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf261ea-efda-4229-b839-f8f14c605895 service cinder] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:03.796643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab510c8-cd44-4cfe-9f7d-fe7bc84b0cc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:03.797199 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab510c8-cd44-4cfe-9f7d-fe7bc84b0cc9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:03.797472 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab510c8-cd44-4cfe-9f7d-fe7bc84b0cc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:03.797734 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab510c8-cd44-4cfe-9f7d-fe7bc84b0cc9 admin admin] PATH_INFO: `/v3/users/2406a55c5be54ed1992cb45616ef2e76` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:03.798925 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab510c8-cd44-4cfe-9f7d-fe7bc84b0cc9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2406a55c5be54ed1992cb45616ef2e76)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:03.800870 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab510c8-cd44-4cfe-9f7d-fe7bc84b0cc9 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:03.852185 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-5ab510c8-cd44-4cfe-9f7d-fe7bc84b0cc9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2406a55c5be54ed1992cb45616ef2e76'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:03.861598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-5ab510c8-cd44-4cfe-9f7d-fe7bc84b0cc9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2406a55c5be54ed1992cb45616ef2e76'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:03.862225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-5ab510c8-cd44-4cfe-9f7d-fe7bc84b0cc9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2406a55c5be54ed1992cb45616ef2e76'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:03.869753 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-5ab510c8-cd44-4cfe-9f7d-fe7bc84b0cc9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2406a55c5be54ed1992cb45616ef2e76'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:03.881583 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-5ab510c8-cd44-4cfe-9f7d-fe7bc84b0cc9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2406a55c5be54ed1992cb45616ef2e76'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:03.887680 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1225/2466] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:09:03 2022] DELETE /identity/v3/users/2406a55c5be54ed1992cb45616ef2e76 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:03.892558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffb63b2-cc2a-4ffd-a6d6-c9673c2c760d None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:03.893071 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffb63b2-cc2a-4ffd-a6d6-c9673c2c760d None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:03.893071 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffb63b2-cc2a-4ffd-a6d6-c9673c2c760d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:03.913253 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-3ffb63b2-cc2a-4ffd-a6d6-c9673c2c760d None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:03.914451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3ffb63b2-cc2a-4ffd-a6d6-c9673c2c760d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:03.947576 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:03.976842 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1242/2467] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:03.986759 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab510c8-cd44-4cfe-9f7d-fe7bc84b0cc9 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:03.988258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-5ab510c8-cd44-4cfe-9f7d-fe7bc84b0cc9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:04.045741 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab510c8-cd44-4cfe-9f7d-fe7bc84b0cc9 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:04.049896 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c5785d-4733-4376-b2dd-5431f48bfd33 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:04.050428 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c5785d-4733-4376-b2dd-5431f48bfd33 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:04.050702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c5785d-4733-4376-b2dd-5431f48bfd33 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:04.050968 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c5785d-4733-4376-b2dd-5431f48bfd33 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:04.060315 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c5785d-4733-4376-b2dd-5431f48bfd33 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:04.061893 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c5785d-4733-4376-b2dd-5431f48bfd33 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:04.071283 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1226/2468] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:09:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 01:09:04.351096 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffb63b2-cc2a-4ffd-a6d6-c9673c2c760d None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:04.352730 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3ffb63b2-cc2a-4ffd-a6d6-c9673c2c760d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:04.372561 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffb63b2-cc2a-4ffd-a6d6-c9673c2c760d None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:04.378193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b540d8fc-88bf-40d1-81d8-e2d8d8a2836b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:04.378839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b540d8fc-88bf-40d1-81d8-e2d8d8a2836b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:04.379111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b540d8fc-88bf-40d1-81d8-e2d8d8a2836b admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:04.379374 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b540d8fc-88bf-40d1-81d8-e2d8d8a2836b admin admin] PATH_INFO: `/v3/projects/510a38bbcf8f47b1838036f68160b1a9` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:04.380609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b540d8fc-88bf-40d1-81d8-e2d8d8a2836b admin admin] RBAC: Authorizing `identity:delete_project(project_id=510a38bbcf8f47b1838036f68160b1a9)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:04.382515 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b540d8fc-88bf-40d1-81d8-e2d8d8a2836b admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:04.438671 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b540d8fc-88bf-40d1-81d8-e2d8d8a2836b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '510a38bbcf8f47b1838036f68160b1a9', 'request_id': 'req-b540d8fc-88bf-40d1-81d8-e2d8d8a2836b'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:04.448063 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b540d8fc-88bf-40d1-81d8-e2d8d8a2836b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '510a38bbcf8f47b1838036f68160b1a9', 'request_id': 'req-b540d8fc-88bf-40d1-81d8-e2d8d8a2836b'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:04.458394 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b540d8fc-88bf-40d1-81d8-e2d8d8a2836b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '510a38bbcf8f47b1838036f68160b1a9', 'request_id': 'req-b540d8fc-88bf-40d1-81d8-e2d8d8a2836b'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:04.471185 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b540d8fc-88bf-40d1-81d8-e2d8d8a2836b admin admin] The token cache is being invalidate because project 510a38bbcf8f47b1838036f68160b1a9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:09:04.471596 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b540d8fc-88bf-40d1-81d8-e2d8d8a2836b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:04.472269 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b540d8fc-88bf-40d1-81d8-e2d8d8a2836b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:04.473766 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1243/2469] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:09:04 2022] DELETE /identity/v3/projects/510a38bbcf8f47b1838036f68160b1a9 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:04.477946 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c5785d-4733-4376-b2dd-5431f48bfd33 service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:04.488282 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c5785d-4733-4376-b2dd-5431f48bfd33 service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:04.507987 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:04.549280 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6f1463-946b-4c49-98b8-d1b1e437f8b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:04.550045 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6f1463-946b-4c49-98b8-d1b1e437f8b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:04.550316 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6f1463-946b-4c49-98b8-d1b1e437f8b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:04.550582 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6f1463-946b-4c49-98b8-d1b1e437f8b1 admin admin] PATH_INFO: `/v3/users/21d0d385bd8a4fa68b98081dc570120d` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:04.551679 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6f1463-946b-4c49-98b8-d1b1e437f8b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=21d0d385bd8a4fa68b98081dc570120d)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:04.553724 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6f1463-946b-4c49-98b8-d1b1e437f8b1 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:04.599880 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-ac6f1463-946b-4c49-98b8-d1b1e437f8b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '21d0d385bd8a4fa68b98081dc570120d'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:04.609398 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-ac6f1463-946b-4c49-98b8-d1b1e437f8b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '21d0d385bd8a4fa68b98081dc570120d'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:04.610071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-ac6f1463-946b-4c49-98b8-d1b1e437f8b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '21d0d385bd8a4fa68b98081dc570120d'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:04.617224 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-ac6f1463-946b-4c49-98b8-d1b1e437f8b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '21d0d385bd8a4fa68b98081dc570120d'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:04.629501 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-ac6f1463-946b-4c49-98b8-d1b1e437f8b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '21d0d385bd8a4fa68b98081dc570120d'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:04.636127 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1227/2470] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:09:04 2022] DELETE /identity/v3/users/21d0d385bd8a4fa68b98081dc570120d => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:04.925152 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b540d8fc-88bf-40d1-81d8-e2d8d8a2836b admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:04.926652 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-b540d8fc-88bf-40d1-81d8-e2d8d8a2836b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:04.941195 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b540d8fc-88bf-40d1-81d8-e2d8d8a2836b admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:04.998641 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07591e61-0133-4efb-b298-b095c0ca479b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:04.999353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07591e61-0133-4efb-b298-b095c0ca479b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:04.999755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07591e61-0133-4efb-b298-b095c0ca479b admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:05.000037 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07591e61-0133-4efb-b298-b095c0ca479b admin admin] PATH_INFO: `/v3/projects/dc0d2c0478fe40abb1d87c868e0f2715` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:05.001132 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07591e61-0133-4efb-b298-b095c0ca479b admin admin] RBAC: Authorizing `identity:delete_project(project_id=dc0d2c0478fe40abb1d87c868e0f2715)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:05.003024 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07591e61-0133-4efb-b298-b095c0ca479b admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:05.057273 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-07591e61-0133-4efb-b298-b095c0ca479b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dc0d2c0478fe40abb1d87c868e0f2715', 'request_id': 'req-07591e61-0133-4efb-b298-b095c0ca479b'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:05.066515 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-07591e61-0133-4efb-b298-b095c0ca479b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dc0d2c0478fe40abb1d87c868e0f2715', 'request_id': 'req-07591e61-0133-4efb-b298-b095c0ca479b'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:05.071294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:05.072781 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-07591e61-0133-4efb-b298-b095c0ca479b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dc0d2c0478fe40abb1d87c868e0f2715', 'request_id': 'req-07591e61-0133-4efb-b298-b095c0ca479b'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:05.086717 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-07591e61-0133-4efb-b298-b095c0ca479b admin admin] The token cache is being invalidate because project dc0d2c0478fe40abb1d87c868e0f2715 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:09:05.087003 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-07591e61-0133-4efb-b298-b095c0ca479b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:05.087667 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-07591e61-0133-4efb-b298-b095c0ca479b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:05.089415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1244/2471] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:09:04 2022] DELETE /identity/v3/projects/dc0d2c0478fe40abb1d87c868e0f2715 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:05.094633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f4b8c5-7ad1-4f9a-9aa7-9117e0530de0 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:05.095144 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f4b8c5-7ad1-4f9a-9aa7-9117e0530de0 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:05.095144 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f4b8c5-7ad1-4f9a-9aa7-9117e0530de0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:05.116515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-47f4b8c5-7ad1-4f9a-9aa7-9117e0530de0 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:05.117645 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-47f4b8c5-7ad1-4f9a-9aa7-9117e0530de0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:05.187066 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1228/2472] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:05.195142 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2508d6-752f-4c7a-b440-159f25fab8bc None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:05.195423 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2508d6-752f-4c7a-b440-159f25fab8bc None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:05.195692 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2508d6-752f-4c7a-b440-159f25fab8bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:05.217320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-0f2508d6-752f-4c7a-b440-159f25fab8bc None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:05.218415 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-0f2508d6-752f-4c7a-b440-159f25fab8bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:05.226742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1245/2473] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:05.231168 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f4b8c5-7ad1-4f9a-9aa7-9117e0530de0 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:05.241231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f4b8c5-7ad1-4f9a-9aa7-9117e0530de0 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:05.245968 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bb7889-eb71-4a02-9935-43e77b5d310a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:05.246597 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bb7889-eb71-4a02-9935-43e77b5d310a admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:05.246872 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bb7889-eb71-4a02-9935-43e77b5d310a admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:05.247141 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bb7889-eb71-4a02-9935-43e77b5d310a admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:05.247827 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bb7889-eb71-4a02-9935-43e77b5d310a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:09:05.248127 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bb7889-eb71-4a02-9935-43e77b5d310a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:05.249752 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bb7889-eb71-4a02-9935-43e77b5d310a admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:05.261981 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1229/2474] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:09:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 01:09:05.266665 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2508d6-752f-4c7a-b440-159f25fab8bc None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:05.276460 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2508d6-752f-4c7a-b440-159f25fab8bc None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:05.281173 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3209a512-52f8-4e71-9fad-b1167ce72ff8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:05.281776 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3209a512-52f8-4e71-9fad-b1167ce72ff8 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:05.282050 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3209a512-52f8-4e71-9fad-b1167ce72ff8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:05.282315 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3209a512-52f8-4e71-9fad-b1167ce72ff8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:05.282952 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3209a512-52f8-4e71-9fad-b1167ce72ff8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:05.284880 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3209a512-52f8-4e71-9fad-b1167ce72ff8 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:05.316663 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1246/2475] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:09:05 2022] POST /identity/v3/projects => generated 365 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:09:05.317891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bb7889-eb71-4a02-9935-43e77b5d310a admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:05.329032 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bb7889-eb71-4a02-9935-43e77b5d310a admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:05.357521 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:05.388856 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b10f028-0890-49db-b728-1f24d00bab06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:05.389367 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b10f028-0890-49db-b728-1f24d00bab06 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:05.389642 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b10f028-0890-49db-b728-1f24d00bab06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:05.390032 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b10f028-0890-49db-b728-1f24d00bab06 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:05.390628 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b10f028-0890-49db-b728-1f24d00bab06 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:05.392594 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b10f028-0890-49db-b728-1f24d00bab06 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:05.418610 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1230/2476] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:09:05 2022] POST /identity/v3/users => generated 283 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:09:05.423244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3209a512-52f8-4e71-9fad-b1167ce72ff8 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:05.434551 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3209a512-52f8-4e71-9fad-b1167ce72ff8 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:05.440039 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834b2e5c-fed1-4092-863b-6b8d70580731 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:05.440648 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834b2e5c-fed1-4092-863b-6b8d70580731 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:05.440926 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834b2e5c-fed1-4092-863b-6b8d70580731 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:05.441190 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834b2e5c-fed1-4092-863b-6b8d70580731 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:05.441840 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834b2e5c-fed1-4092-863b-6b8d70580731 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:05.443442 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834b2e5c-fed1-4092-863b-6b8d70580731 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:05.453863 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1247/2477] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:05 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:09:05.457987 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b10f028-0890-49db-b728-1f24d00bab06 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:05.468734 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b10f028-0890-49db-b728-1f24d00bab06 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:05.473583 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc58082-feb3-4719-a3f4-c312d4955811 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:05.474254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc58082-feb3-4719-a3f4-c312d4955811 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:05.474522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc58082-feb3-4719-a3f4-c312d4955811 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:05.474788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc58082-feb3-4719-a3f4-c312d4955811 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:05.475430 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc58082-feb3-4719-a3f4-c312d4955811 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:05.477114 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc58082-feb3-4719-a3f4-c312d4955811 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:05.488854 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1231/2478] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:05 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:09:05.493246 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834b2e5c-fed1-4092-863b-6b8d70580731 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:05.504050 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834b2e5c-fed1-4092-863b-6b8d70580731 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:05.509881 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c287e5-3ee3-40ab-9805-54d4aaf9652b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:05.510536 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c287e5-3ee3-40ab-9805-54d4aaf9652b admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:05.510796 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c287e5-3ee3-40ab-9805-54d4aaf9652b admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:05.511063 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c287e5-3ee3-40ab-9805-54d4aaf9652b admin admin] PATH_INFO: `/v3/projects/183924fe30e949919847fc51afdec2d8/users/b6e4736f7d384341b8b222f0b2665f01/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:05.526704 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c287e5-3ee3-40ab-9805-54d4aaf9652b admin admin] RBAC: Authorizing `identity:create_grant(project_id=183924fe30e949919847fc51afdec2d8, user_id=b6e4736f7d384341b8b222f0b2665f01, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:05.529432 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c287e5-3ee3-40ab-9805-54d4aaf9652b admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:05.544927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1248/2479] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:09:05 2022] PUT /identity/v3/projects/183924fe30e949919847fc51afdec2d8/users/b6e4736f7d384341b8b222f0b2665f01/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:05.555781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e878e708-1823-4993-8d98-b956f197e3e2 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:05.556085 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e878e708-1823-4993-8d98-b956f197e3e2 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:05.556354 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e878e708-1823-4993-8d98-b956f197e3e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:05.585986 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-e878e708-1823-4993-8d98-b956f197e3e2 None None] MFA Rules not processed for user `b6e4736f7d384341b8b222f0b2665f01`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:05.587188 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-e878e708-1823-4993-8d98-b956f197e3e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:05.657519 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1232/2480] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:05 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:05.662236 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c287e5-3ee3-40ab-9805-54d4aaf9652b admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:05.726036 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c287e5-3ee3-40ab-9805-54d4aaf9652b admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:05.731276 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10705b68-05f8-4d6d-9b64-e7074190508e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:05.731912 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10705b68-05f8-4d6d-9b64-e7074190508e admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:05.732182 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10705b68-05f8-4d6d-9b64-e7074190508e admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:05.732446 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10705b68-05f8-4d6d-9b64-e7074190508e admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:05.733059 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10705b68-05f8-4d6d-9b64-e7074190508e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:05.735119 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10705b68-05f8-4d6d-9b64-e7074190508e admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:05.766994 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1249/2481] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:09:05 2022] POST /identity/v3/projects => generated 365 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:09:05.769237 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e878e708-1823-4993-8d98-b956f197e3e2 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:05.780340 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e878e708-1823-4993-8d98-b956f197e3e2 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:05.838996 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dede0a62-4612-44d0-8389-1fd0b55ff21a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:05.839617 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dede0a62-4612-44d0-8389-1fd0b55ff21a admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:05.839916 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dede0a62-4612-44d0-8389-1fd0b55ff21a admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:05.840186 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dede0a62-4612-44d0-8389-1fd0b55ff21a admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:05.840821 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dede0a62-4612-44d0-8389-1fd0b55ff21a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:05.842766 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dede0a62-4612-44d0-8389-1fd0b55ff21a admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:05.866736 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1233/2482] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:09:05 2022] POST /identity/v3/users => generated 289 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:09:05.870935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10705b68-05f8-4d6d-9b64-e7074190508e admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:05.881916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10705b68-05f8-4d6d-9b64-e7074190508e admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:05.887294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c25345-eb99-4234-8672-83e71f66f717 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:05.887932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c25345-eb99-4234-8672-83e71f66f717 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:05.888208 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c25345-eb99-4234-8672-83e71f66f717 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:05.888473 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c25345-eb99-4234-8672-83e71f66f717 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:05.889108 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c25345-eb99-4234-8672-83e71f66f717 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:05.890911 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c25345-eb99-4234-8672-83e71f66f717 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:05.904947 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1250/2483] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:05 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:09:05.906197 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dede0a62-4612-44d0-8389-1fd0b55ff21a admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:05.916752 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dede0a62-4612-44d0-8389-1fd0b55ff21a admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:05.921570 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8658d1-7d39-4199-be00-648fa30c00f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:05.922224 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8658d1-7d39-4199-be00-648fa30c00f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:05.922494 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8658d1-7d39-4199-be00-648fa30c00f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:05.922757 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8658d1-7d39-4199-be00-648fa30c00f9 admin admin] PATH_INFO: `/v3/projects/98cb66590aa1442fbc87e29a891159ed/users/a57b9efd3074463abe08e437fc3d49db/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:05.939074 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8658d1-7d39-4199-be00-648fa30c00f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=98cb66590aa1442fbc87e29a891159ed, user_id=a57b9efd3074463abe08e437fc3d49db, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:05.941818 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8658d1-7d39-4199-be00-648fa30c00f9 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:05.956666 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1234/2484] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:09:05 2022] PUT /identity/v3/projects/98cb66590aa1442fbc87e29a891159ed/users/a57b9efd3074463abe08e437fc3d49db/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:05.966477 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c25345-eb99-4234-8672-83e71f66f717 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:05.975026 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8658d1-7d39-4199-be00-648fa30c00f9 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:05.976829 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-ff8658d1-7d39-4199-be00-648fa30c00f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:05.977430 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c25345-eb99-4234-8672-83e71f66f717 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:05.982155 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:06.037346 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289ba0f0-67f8-4e88-99f7-42bf9d2fc84a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:06.037956 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289ba0f0-67f8-4e88-99f7-42bf9d2fc84a admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:06.038256 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289ba0f0-67f8-4e88-99f7-42bf9d2fc84a admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:06.038526 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289ba0f0-67f8-4e88-99f7-42bf9d2fc84a admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:06.039165 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289ba0f0-67f8-4e88-99f7-42bf9d2fc84a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:06.039274 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8658d1-7d39-4199-be00-648fa30c00f9 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:06.040861 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289ba0f0-67f8-4e88-99f7-42bf9d2fc84a admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:06.042932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:06.043098 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411c474f-01bb-4211-aeae-560e921bc875 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:06.043791 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411c474f-01bb-4211-aeae-560e921bc875 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:06.044064 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411c474f-01bb-4211-aeae-560e921bc875 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:06.044356 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411c474f-01bb-4211-aeae-560e921bc875 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:06.046357 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-411c474f-01bb-4211-aeae-560e921bc875 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:06.055110 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1251/2485] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:05 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:09:06.056288 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289ba0f0-67f8-4e88-99f7-42bf9d2fc84a admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:06.066202 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289ba0f0-67f8-4e88-99f7-42bf9d2fc84a admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:06.070787 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5bdcb1-3f15-4806-8e56-1fc677037fc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:06.071430 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5bdcb1-3f15-4806-8e56-1fc677037fc7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:06.071703 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5bdcb1-3f15-4806-8e56-1fc677037fc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:06.072008 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5bdcb1-3f15-4806-8e56-1fc677037fc7 admin admin] PATH_INFO: `/v3/projects/98cb66590aa1442fbc87e29a891159ed/users/a57b9efd3074463abe08e437fc3d49db/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:06.073922 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5bdcb1-3f15-4806-8e56-1fc677037fc7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=98cb66590aa1442fbc87e29a891159ed, user_id=a57b9efd3074463abe08e437fc3d49db, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:06.076643 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5bdcb1-3f15-4806-8e56-1fc677037fc7 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:06.089579 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1252/2486] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:09:06 2022] PUT /identity/v3/projects/98cb66590aa1442fbc87e29a891159ed/users/a57b9efd3074463abe08e437fc3d49db/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:06.101389 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bffb62-ca09-4219-aa1f-220fbfd2ec3e None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:06.101665 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bffb62-ca09-4219-aa1f-220fbfd2ec3e None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:06.101966 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bffb62-ca09-4219-aa1f-220fbfd2ec3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:06.107975 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411c474f-01bb-4211-aeae-560e921bc875 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:06.109642 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411c474f-01bb-4211-aeae-560e921bc875 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:06.118962 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1235/2487] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:09:05 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 01:09:06.133143 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-b6bffb62-ca09-4219-aa1f-220fbfd2ec3e None None] MFA Rules not processed for user `a57b9efd3074463abe08e437fc3d49db`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:06.134358 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-b6bffb62-ca09-4219-aa1f-220fbfd2ec3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:06.217360 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1253/2488] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:06 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:09:06.229226 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411c474f-01bb-4211-aeae-560e921bc875 service cinder] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:06.230572 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-411c474f-01bb-4211-aeae-560e921bc875 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:06.241691 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411c474f-01bb-4211-aeae-560e921bc875 service cinder] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:06.296764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3ab440-ff36-4692-8047-97052dbe44fd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:06.297378 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3ab440-ff36-4692-8047-97052dbe44fd service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:06.297655 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3ab440-ff36-4692-8047-97052dbe44fd service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:06.297920 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3ab440-ff36-4692-8047-97052dbe44fd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:06.307522 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3ab440-ff36-4692-8047-97052dbe44fd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:06.309033 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3ab440-ff36-4692-8047-97052dbe44fd service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:06.320129 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1236/2489] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:09:06 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:09:06.406983 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6bffb62-ca09-4219-aa1f-220fbfd2ec3e None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:06.417299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6bffb62-ca09-4219-aa1f-220fbfd2ec3e None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:06.422806 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2b4e07-a553-4045-ba38-ccaf4e8f5319 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:06.423421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2b4e07-a553-4045-ba38-ccaf4e8f5319 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:06.423692 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2b4e07-a553-4045-ba38-ccaf4e8f5319 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:06.423990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2b4e07-a553-4045-ba38-ccaf4e8f5319 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:06.485907 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2b4e07-a553-4045-ba38-ccaf4e8f5319 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:06.487639 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2b4e07-a553-4045-ba38-ccaf4e8f5319 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:06.497778 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1254/2490] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:09:06 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:09:08.360446 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3ab440-ff36-4692-8047-97052dbe44fd service cinder] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:08.362279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-2c3ab440-ff36-4692-8047-97052dbe44fd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:08.429116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3ab440-ff36-4692-8047-97052dbe44fd service cinder] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:08.433544 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ce9013-b467-4643-a837-2012a3670871 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:08.434177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ce9013-b467-4643-a837-2012a3670871 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:08.434447 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ce9013-b467-4643-a837-2012a3670871 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:08.434738 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ce9013-b467-4643-a837-2012a3670871 admin admin] PATH_INFO: `/v3/users/11e58f202d8a4249b2905961b1140b99` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:08.435868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ce9013-b467-4643-a837-2012a3670871 admin admin] RBAC: Authorizing `identity:delete_user(user_id=11e58f202d8a4249b2905961b1140b99)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:08.437879 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ce9013-b467-4643-a837-2012a3670871 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:08.439822 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=100 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:08.491844 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-05ce9013-b467-4643-a837-2012a3670871 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '11e58f202d8a4249b2905961b1140b99'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:08.502411 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-05ce9013-b467-4643-a837-2012a3670871 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '11e58f202d8a4249b2905961b1140b99'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:08.503070 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-05ce9013-b467-4643-a837-2012a3670871 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '11e58f202d8a4249b2905961b1140b99'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:08.511026 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-05ce9013-b467-4643-a837-2012a3670871 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '11e58f202d8a4249b2905961b1140b99'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:08.523842 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-05ce9013-b467-4643-a837-2012a3670871 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '11e58f202d8a4249b2905961b1140b99'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:08.530533 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1237/2491] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:09:08 2022] DELETE /identity/v3/users/11e58f202d8a4249b2905961b1140b99 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:08.535464 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d4559a-a3e3-4f18-b8e2-cb98f295418c None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:08.536025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d4559a-a3e3-4f18-b8e2-cb98f295418c None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:08.536131 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d4559a-a3e3-4f18-b8e2-cb98f295418c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:08.557499 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-26d4559a-a3e3-4f18-b8e2-cb98f295418c None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:08.558698 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-26d4559a-a3e3-4f18-b8e2-cb98f295418c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:08.617494 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1255/2492] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:08.627470 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ce9013-b467-4643-a837-2012a3670871 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:08.628999 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-05ce9013-b467-4643-a837-2012a3670871 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:08.687443 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ce9013-b467-4643-a837-2012a3670871 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:08.691613 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9dda6a-8656-4872-9068-3971a95f18b9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:08.692254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9dda6a-8656-4872-9068-3971a95f18b9 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:08.692534 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9dda6a-8656-4872-9068-3971a95f18b9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:08.692799 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9dda6a-8656-4872-9068-3971a95f18b9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:08.702367 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9dda6a-8656-4872-9068-3971a95f18b9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:08.703805 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9dda6a-8656-4872-9068-3971a95f18b9 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:08.713852 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1238/2493] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:09:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:09:08.987438 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d4559a-a3e3-4f18-b8e2-cb98f295418c None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:08.989063 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-26d4559a-a3e3-4f18-b8e2-cb98f295418c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:09.002870 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d4559a-a3e3-4f18-b8e2-cb98f295418c None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:09.008590 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000b616f-4500-4c1d-b890-7ec5dbec89c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:09.009404 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000b616f-4500-4c1d-b890-7ec5dbec89c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:09.009780 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000b616f-4500-4c1d-b890-7ec5dbec89c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:09.010211 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000b616f-4500-4c1d-b890-7ec5dbec89c4 admin admin] PATH_INFO: `/v3/projects/f7745fb95e6b4956a08b711aeec7832b` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:09.011517 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000b616f-4500-4c1d-b890-7ec5dbec89c4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7745fb95e6b4956a08b711aeec7832b)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:09.013472 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000b616f-4500-4c1d-b890-7ec5dbec89c4 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:09.015979 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:09.062391 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-000b616f-4500-4c1d-b890-7ec5dbec89c4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7745fb95e6b4956a08b711aeec7832b', 'request_id': 'req-000b616f-4500-4c1d-b890-7ec5dbec89c4'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:09.070928 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-000b616f-4500-4c1d-b890-7ec5dbec89c4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7745fb95e6b4956a08b711aeec7832b', 'request_id': 'req-000b616f-4500-4c1d-b890-7ec5dbec89c4'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:09.076652 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-000b616f-4500-4c1d-b890-7ec5dbec89c4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7745fb95e6b4956a08b711aeec7832b', 'request_id': 'req-000b616f-4500-4c1d-b890-7ec5dbec89c4'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:09.088122 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-000b616f-4500-4c1d-b890-7ec5dbec89c4 admin admin] The token cache is being invalidate because project f7745fb95e6b4956a08b711aeec7832b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:09:09.088398 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-000b616f-4500-4c1d-b890-7ec5dbec89c4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:09.089018 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-000b616f-4500-4c1d-b890-7ec5dbec89c4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:09.090515 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1256/2494] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:09:08 2022] DELETE /identity/v3/projects/f7745fb95e6b4956a08b711aeec7832b => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:09.094752 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9dda6a-8656-4872-9068-3971a95f18b9 service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:09.104661 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9dda6a-8656-4872-9068-3971a95f18b9 service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:09.165324 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c38fa0-3c48-4737-99db-77d576784f4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:09.165995 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c38fa0-3c48-4737-99db-77d576784f4c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:09.166261 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c38fa0-3c48-4737-99db-77d576784f4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:09.166525 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c38fa0-3c48-4737-99db-77d576784f4c admin admin] PATH_INFO: `/v3/users/802474ffacbb40d9b202d187a439fe55` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:09.167819 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c38fa0-3c48-4737-99db-77d576784f4c admin admin] RBAC: Authorizing `identity:delete_user(user_id=802474ffacbb40d9b202d187a439fe55)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:09.169707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c38fa0-3c48-4737-99db-77d576784f4c admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:09.192666 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:09.220468 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-32c38fa0-3c48-4737-99db-77d576784f4c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '802474ffacbb40d9b202d187a439fe55'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:09.229446 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-32c38fa0-3c48-4737-99db-77d576784f4c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '802474ffacbb40d9b202d187a439fe55'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:09.230070 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-32c38fa0-3c48-4737-99db-77d576784f4c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '802474ffacbb40d9b202d187a439fe55'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:09.236950 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-32c38fa0-3c48-4737-99db-77d576784f4c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '802474ffacbb40d9b202d187a439fe55'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:09.248525 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-32c38fa0-3c48-4737-99db-77d576784f4c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '802474ffacbb40d9b202d187a439fe55'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:09.255407 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1239/2495] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:09:09 2022] DELETE /identity/v3/users/802474ffacbb40d9b202d187a439fe55 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:09.551005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000b616f-4500-4c1d-b890-7ec5dbec89c4 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:09.552318 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-000b616f-4500-4c1d-b890-7ec5dbec89c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:09.567522 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000b616f-4500-4c1d-b890-7ec5dbec89c4 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:09.637629 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e2a5e5-db4d-4f91-8041-ec950145e285 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:09.638282 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e2a5e5-db4d-4f91-8041-ec950145e285 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:09.638556 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e2a5e5-db4d-4f91-8041-ec950145e285 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:09.638823 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e2a5e5-db4d-4f91-8041-ec950145e285 admin admin] PATH_INFO: `/v3/projects/7b347736df4b4907adc4e112f9c81aa6` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:09.640001 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e2a5e5-db4d-4f91-8041-ec950145e285 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7b347736df4b4907adc4e112f9c81aa6)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:09.642143 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e2a5e5-db4d-4f91-8041-ec950145e285 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:09.692296 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-89e2a5e5-db4d-4f91-8041-ec950145e285 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7b347736df4b4907adc4e112f9c81aa6', 'request_id': 'req-89e2a5e5-db4d-4f91-8041-ec950145e285'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:09.701324 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-89e2a5e5-db4d-4f91-8041-ec950145e285 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7b347736df4b4907adc4e112f9c81aa6', 'request_id': 'req-89e2a5e5-db4d-4f91-8041-ec950145e285'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:09.707595 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-89e2a5e5-db4d-4f91-8041-ec950145e285 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7b347736df4b4907adc4e112f9c81aa6', 'request_id': 'req-89e2a5e5-db4d-4f91-8041-ec950145e285'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:09.720025 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-89e2a5e5-db4d-4f91-8041-ec950145e285 admin admin] The token cache is being invalidate because project 7b347736df4b4907adc4e112f9c81aa6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:09:09.720303 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-89e2a5e5-db4d-4f91-8041-ec950145e285 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:09.720962 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-89e2a5e5-db4d-4f91-8041-ec950145e285 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:09.722469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1257/2496] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:09:09 2022] DELETE /identity/v3/projects/7b347736df4b4907adc4e112f9c81aa6 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:09.727064 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c38fa0-3c48-4737-99db-77d576784f4c admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:09.728755 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-32c38fa0-3c48-4737-99db-77d576784f4c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:09.742341 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c38fa0-3c48-4737-99db-77d576784f4c admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:09.797896 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251bce16-c7d2-4cd1-a7a5-28711bc0f832 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:09.798619 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251bce16-c7d2-4cd1-a7a5-28711bc0f832 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:09.798917 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251bce16-c7d2-4cd1-a7a5-28711bc0f832 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:09.799180 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251bce16-c7d2-4cd1-a7a5-28711bc0f832 admin admin] PATH_INFO: `/v3/users/9817e4a34e3745c0bbd638968cadacc7` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:09.800330 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251bce16-c7d2-4cd1-a7a5-28711bc0f832 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9817e4a34e3745c0bbd638968cadacc7)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:09.802239 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251bce16-c7d2-4cd1-a7a5-28711bc0f832 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:09.853723 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-251bce16-c7d2-4cd1-a7a5-28711bc0f832 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9817e4a34e3745c0bbd638968cadacc7'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:09.862043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-251bce16-c7d2-4cd1-a7a5-28711bc0f832 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9817e4a34e3745c0bbd638968cadacc7'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:09.862751 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-251bce16-c7d2-4cd1-a7a5-28711bc0f832 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9817e4a34e3745c0bbd638968cadacc7'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:09.870511 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-251bce16-c7d2-4cd1-a7a5-28711bc0f832 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9817e4a34e3745c0bbd638968cadacc7'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:09.886475 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-251bce16-c7d2-4cd1-a7a5-28711bc0f832 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9817e4a34e3745c0bbd638968cadacc7'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:09.892574 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1240/2497] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:09:09 2022] DELETE /identity/v3/users/9817e4a34e3745c0bbd638968cadacc7 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:10.185012 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e2a5e5-db4d-4f91-8041-ec950145e285 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:10.186427 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-89e2a5e5-db4d-4f91-8041-ec950145e285 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:10.199342 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e2a5e5-db4d-4f91-8041-ec950145e285 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:10.220033 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:10.259293 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a2e1de-46ce-4473-b062-ae946a031641 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:10.260001 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a2e1de-46ce-4473-b062-ae946a031641 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:10.260275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a2e1de-46ce-4473-b062-ae946a031641 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:10.260537 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a2e1de-46ce-4473-b062-ae946a031641 admin admin] PATH_INFO: `/v3/projects/53cdbef41fdf42bd83b7ca286b22d97d` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:10.261660 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a2e1de-46ce-4473-b062-ae946a031641 admin admin] RBAC: Authorizing `identity:delete_project(project_id=53cdbef41fdf42bd83b7ca286b22d97d)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:10.263540 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a2e1de-46ce-4473-b062-ae946a031641 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:10.315512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a3a2e1de-46ce-4473-b062-ae946a031641 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '53cdbef41fdf42bd83b7ca286b22d97d', 'request_id': 'req-a3a2e1de-46ce-4473-b062-ae946a031641'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:10.327916 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a3a2e1de-46ce-4473-b062-ae946a031641 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '53cdbef41fdf42bd83b7ca286b22d97d', 'request_id': 'req-a3a2e1de-46ce-4473-b062-ae946a031641'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:10.334792 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a3a2e1de-46ce-4473-b062-ae946a031641 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '53cdbef41fdf42bd83b7ca286b22d97d', 'request_id': 'req-a3a2e1de-46ce-4473-b062-ae946a031641'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:10.346935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a3a2e1de-46ce-4473-b062-ae946a031641 admin admin] The token cache is being invalidate because project 53cdbef41fdf42bd83b7ca286b22d97d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:09:10.346935 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a3a2e1de-46ce-4473-b062-ae946a031641 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:10.347219 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a3a2e1de-46ce-4473-b062-ae946a031641 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:10.348849 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1258/2498] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:09:10 2022] DELETE /identity/v3/projects/53cdbef41fdf42bd83b7ca286b22d97d => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:10.355251 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69fffa0-2cfc-40c5-a4fb-8aaf3662957f None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:10.355526 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69fffa0-2cfc-40c5-a4fb-8aaf3662957f None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:10.355816 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69fffa0-2cfc-40c5-a4fb-8aaf3662957f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:10.379933 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-f69fffa0-2cfc-40c5-a4fb-8aaf3662957f None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:10.381184 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-f69fffa0-2cfc-40c5-a4fb-8aaf3662957f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:10.395375 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:10.449254 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1241/2499] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:10.459704 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa48a99f-45ad-400c-b27e-15bb71e05f2f None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:10.460018 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa48a99f-45ad-400c-b27e-15bb71e05f2f None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:10.460287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa48a99f-45ad-400c-b27e-15bb71e05f2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:10.482287 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-fa48a99f-45ad-400c-b27e-15bb71e05f2f None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:10.483524 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-fa48a99f-45ad-400c-b27e-15bb71e05f2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:10.492657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1259/2500] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:10.497294 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69fffa0-2cfc-40c5-a4fb-8aaf3662957f None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:10.507747 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69fffa0-2cfc-40c5-a4fb-8aaf3662957f None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:10.513407 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d21a9c-9743-4ac9-8c42-9f4dc2ef6152 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:10.514018 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d21a9c-9743-4ac9-8c42-9f4dc2ef6152 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:10.514285 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d21a9c-9743-4ac9-8c42-9f4dc2ef6152 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:10.514551 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d21a9c-9743-4ac9-8c42-9f4dc2ef6152 admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:10.515219 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d21a9c-9743-4ac9-8c42-9f4dc2ef6152 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:09:10.515508 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d21a9c-9743-4ac9-8c42-9f4dc2ef6152 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:10.517132 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d21a9c-9743-4ac9-8c42-9f4dc2ef6152 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:10.529407 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1242/2501] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:09:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 01:09:10.534182 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa48a99f-45ad-400c-b27e-15bb71e05f2f None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:10.543820 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa48a99f-45ad-400c-b27e-15bb71e05f2f None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:10.549697 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d86529a-1865-4ea0-9324-1df44d35f412 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:10.550295 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d86529a-1865-4ea0-9324-1df44d35f412 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:10.550563 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d86529a-1865-4ea0-9324-1df44d35f412 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:10.550826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d86529a-1865-4ea0-9324-1df44d35f412 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:10.551440 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d86529a-1865-4ea0-9324-1df44d35f412 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:10.553364 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d86529a-1865-4ea0-9324-1df44d35f412 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:10.586275 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1260/2502] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:09:10 2022] POST /identity/v3/projects => generated 379 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:09:10.590984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d21a9c-9743-4ac9-8c42-9f4dc2ef6152 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:10.603207 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d21a9c-9743-4ac9-8c42-9f4dc2ef6152 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:10.659599 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa288d8-d572-411a-bb1e-92c3569bb9bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:10.660278 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa288d8-d572-411a-bb1e-92c3569bb9bb admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:10.660594 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa288d8-d572-411a-bb1e-92c3569bb9bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:10.660843 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa288d8-d572-411a-bb1e-92c3569bb9bb admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:10.661470 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa288d8-d572-411a-bb1e-92c3569bb9bb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:10.663388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa288d8-d572-411a-bb1e-92c3569bb9bb admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:10.689350 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1243/2503] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:09:10 2022] POST /identity/v3/users => generated 290 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:09:10.693548 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d86529a-1865-4ea0-9324-1df44d35f412 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:10.704501 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d86529a-1865-4ea0-9324-1df44d35f412 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:10.710426 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7b1572-1751-47df-9002-6e397d515e73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:10.711065 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7b1572-1751-47df-9002-6e397d515e73 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:10.711344 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7b1572-1751-47df-9002-6e397d515e73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:10.711612 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7b1572-1751-47df-9002-6e397d515e73 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:10.712295 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7b1572-1751-47df-9002-6e397d515e73 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:10.714227 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b7b1572-1751-47df-9002-6e397d515e73 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:10.725095 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1261/2504] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:10 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:09:10.729633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa288d8-d572-411a-bb1e-92c3569bb9bb admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:10.745607 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa288d8-d572-411a-bb1e-92c3569bb9bb admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:10.753579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ab23d6-c5de-4895-a4f1-dc8771627f92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:10.754178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ab23d6-c5de-4895-a4f1-dc8771627f92 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:10.754453 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ab23d6-c5de-4895-a4f1-dc8771627f92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:10.754723 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ab23d6-c5de-4895-a4f1-dc8771627f92 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:10.755360 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ab23d6-c5de-4895-a4f1-dc8771627f92 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:10.757013 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ab23d6-c5de-4895-a4f1-dc8771627f92 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:10.768586 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1244/2505] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:10 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:09:10.775310 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7b1572-1751-47df-9002-6e397d515e73 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:10.788598 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7b1572-1751-47df-9002-6e397d515e73 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:10.794517 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03a205c-43ac-4647-9490-0442045240e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:10.795166 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03a205c-43ac-4647-9490-0442045240e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:10.795441 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03a205c-43ac-4647-9490-0442045240e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:10.795711 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03a205c-43ac-4647-9490-0442045240e7 admin admin] PATH_INFO: `/v3/projects/6c7a42e7e9ae438da36e6ac6ed9d4712/users/f45fa00e4e6144f79724f9d542cf3c87/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:10.813816 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03a205c-43ac-4647-9490-0442045240e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6c7a42e7e9ae438da36e6ac6ed9d4712, user_id=f45fa00e4e6144f79724f9d542cf3c87, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:10.816584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03a205c-43ac-4647-9490-0442045240e7 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:10.832032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1262/2506] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:09:10 2022] PUT /identity/v3/projects/6c7a42e7e9ae438da36e6ac6ed9d4712/users/f45fa00e4e6144f79724f9d542cf3c87/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:10.844892 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230c2f73-d375-429a-b187-c4dfba7042e4 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:10.845338 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230c2f73-d375-429a-b187-c4dfba7042e4 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:10.845626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230c2f73-d375-429a-b187-c4dfba7042e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:10.878963 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-230c2f73-d375-429a-b187-c4dfba7042e4 None None] MFA Rules not processed for user `f45fa00e4e6144f79724f9d542cf3c87`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:10.880867 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-230c2f73-d375-429a-b187-c4dfba7042e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:10.952129 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1245/2507] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:10 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:10.958658 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03a205c-43ac-4647-9490-0442045240e7 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:11.008358 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:11.027028 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03a205c-43ac-4647-9490-0442045240e7 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:11.032333 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f300480-c198-4520-b55a-53996d306b0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:11.032975 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f300480-c198-4520-b55a-53996d306b0a admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:11.033244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f300480-c198-4520-b55a-53996d306b0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:11.033506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f300480-c198-4520-b55a-53996d306b0a admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:11.034142 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f300480-c198-4520-b55a-53996d306b0a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:11.036350 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f300480-c198-4520-b55a-53996d306b0a admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:11.064885 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1263/2508] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:09:10 2022] POST /identity/v3/projects => generated 379 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:09:11.069191 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230c2f73-d375-429a-b187-c4dfba7042e4 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:11.081220 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230c2f73-d375-429a-b187-c4dfba7042e4 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:11.093845 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:11.140156 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8848b5f5-22fb-448b-9b81-ab099d5a9587 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:11.140765 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8848b5f5-22fb-448b-9b81-ab099d5a9587 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:11.141036 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8848b5f5-22fb-448b-9b81-ab099d5a9587 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:11.141302 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8848b5f5-22fb-448b-9b81-ab099d5a9587 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:11.141927 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8848b5f5-22fb-448b-9b81-ab099d5a9587 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:11.143906 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8848b5f5-22fb-448b-9b81-ab099d5a9587 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:11.182208 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1246/2509] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:09:11 2022] POST /identity/v3/users => generated 290 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:09:11.183451 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f300480-c198-4520-b55a-53996d306b0a admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:11.194716 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f300480-c198-4520-b55a-53996d306b0a admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:11.199523 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9b201d-b0da-4fd0-8297-688e60bff8b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:11.200159 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9b201d-b0da-4fd0-8297-688e60bff8b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:11.200465 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9b201d-b0da-4fd0-8297-688e60bff8b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:11.200839 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9b201d-b0da-4fd0-8297-688e60bff8b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:11.201498 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9b201d-b0da-4fd0-8297-688e60bff8b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:11.203291 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9b201d-b0da-4fd0-8297-688e60bff8b2 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:11.215057 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1264/2510] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:11 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:09:11.219228 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8848b5f5-22fb-448b-9b81-ab099d5a9587 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:11.230060 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8848b5f5-22fb-448b-9b81-ab099d5a9587 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:11.234689 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75150038-beac-4080-9bff-6ef730b3c431 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:11.235283 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75150038-beac-4080-9bff-6ef730b3c431 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:11.235554 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75150038-beac-4080-9bff-6ef730b3c431 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:11.235866 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75150038-beac-4080-9bff-6ef730b3c431 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:11.236491 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75150038-beac-4080-9bff-6ef730b3c431 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:11.238194 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75150038-beac-4080-9bff-6ef730b3c431 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:11.248855 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1247/2511] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:11 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:09:11.253167 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9b201d-b0da-4fd0-8297-688e60bff8b2 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:11.265630 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9b201d-b0da-4fd0-8297-688e60bff8b2 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:11.272063 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd523a89-cd07-42a8-ac73-e6d0bd675324 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:11.272753 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd523a89-cd07-42a8-ac73-e6d0bd675324 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:11.273020 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd523a89-cd07-42a8-ac73-e6d0bd675324 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:11.273333 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd523a89-cd07-42a8-ac73-e6d0bd675324 admin admin] PATH_INFO: `/v3/projects/541e27b1868841b8b4ba09030ea67bfe/users/9e9a69578c6546b99a67073a2a033452/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:11.289590 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd523a89-cd07-42a8-ac73-e6d0bd675324 admin admin] RBAC: Authorizing `identity:create_grant(project_id=541e27b1868841b8b4ba09030ea67bfe, user_id=9e9a69578c6546b99a67073a2a033452, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:11.292312 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd523a89-cd07-42a8-ac73-e6d0bd675324 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:11.307645 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1265/2512] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:09:11 2022] PUT /identity/v3/projects/541e27b1868841b8b4ba09030ea67bfe/users/9e9a69578c6546b99a67073a2a033452/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:09:11.319913 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318b2d28-49ab-448d-8a29-a258ddd20d0e None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:11.319913 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318b2d28-49ab-448d-8a29-a258ddd20d0e None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:11.319913 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318b2d28-49ab-448d-8a29-a258ddd20d0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:11.338027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd523a89-cd07-42a8-ac73-e6d0bd675324 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:11.339742 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-fd523a89-cd07-42a8-ac73-e6d0bd675324 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:11.350961 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-318b2d28-49ab-448d-8a29-a258ddd20d0e None None] MFA Rules not processed for user `9e9a69578c6546b99a67073a2a033452`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:11.352211 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-318b2d28-49ab-448d-8a29-a258ddd20d0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:11.406851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd523a89-cd07-42a8-ac73-e6d0bd675324 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:11.412341 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8016235b-ac62-4f7c-b38b-dae5ba920eb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:11.413005 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8016235b-ac62-4f7c-b38b-dae5ba920eb0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:11.413273 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8016235b-ac62-4f7c-b38b-dae5ba920eb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:11.413564 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8016235b-ac62-4f7c-b38b-dae5ba920eb0 admin admin] PATH_INFO: `/v3/users/b6e4736f7d384341b8b222f0b2665f01` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:11.414701 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8016235b-ac62-4f7c-b38b-dae5ba920eb0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b6e4736f7d384341b8b222f0b2665f01)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:11.416718 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8016235b-ac62-4f7c-b38b-dae5ba920eb0 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:11.420306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1248/2513] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:11 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:11.424712 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318b2d28-49ab-448d-8a29-a258ddd20d0e None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:11.435949 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318b2d28-49ab-448d-8a29-a258ddd20d0e None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:11.440520 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a80c97-7534-4f45-967a-905f633e17da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:11.441521 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a80c97-7534-4f45-967a-905f633e17da admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:11.441838 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a80c97-7534-4f45-967a-905f633e17da admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:11.442154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a80c97-7534-4f45-967a-905f633e17da admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:11.442862 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a80c97-7534-4f45-967a-905f633e17da admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:11.445020 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a80c97-7534-4f45-967a-905f633e17da admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:11.467230 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8016235b-ac62-4f7c-b38b-dae5ba920eb0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b6e4736f7d384341b8b222f0b2665f01'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:11.474586 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8016235b-ac62-4f7c-b38b-dae5ba920eb0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b6e4736f7d384341b8b222f0b2665f01'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:11.475184 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1249/2514] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:09:11 2022] POST /identity/v3/projects => generated 377 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:09:11.475773 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8016235b-ac62-4f7c-b38b-dae5ba920eb0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b6e4736f7d384341b8b222f0b2665f01'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:11.479132 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a80c97-7534-4f45-967a-905f633e17da admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:11.481053 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-90a80c97-7534-4f45-967a-905f633e17da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:11.489711 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8016235b-ac62-4f7c-b38b-dae5ba920eb0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b6e4736f7d384341b8b222f0b2665f01'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:11.496883 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-8016235b-ac62-4f7c-b38b-dae5ba920eb0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b6e4736f7d384341b8b222f0b2665f01'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:11.503000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1266/2515] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:09:11 2022] DELETE /identity/v3/users/b6e4736f7d384341b8b222f0b2665f01 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:09:11.507407 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b688be5-d851-4684-9f54-dcf1aa42f5ef None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:11.507749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b688be5-d851-4684-9f54-dcf1aa42f5ef None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:11.508041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b688be5-d851-4684-9f54-dcf1aa42f5ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:11.528059 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-2b688be5-d851-4684-9f54-dcf1aa42f5ef None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:11.529272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-2b688be5-d851-4684-9f54-dcf1aa42f5ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:11.541441 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:11.545887 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a80c97-7534-4f45-967a-905f633e17da admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:11.551109 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:11.591121 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1267/2516] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:09:11.601388 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b688be5-d851-4684-9f54-dcf1aa42f5ef None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:11.602803 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-2b688be5-d851-4684-9f54-dcf1aa42f5ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:11.603485 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b2b88a-cce3-47b3-87a7-acaeb6434376 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:11.604137 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b2b88a-cce3-47b3-87a7-acaeb6434376 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:11.604404 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b2b88a-cce3-47b3-87a7-acaeb6434376 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:11.604666 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b2b88a-cce3-47b3-87a7-acaeb6434376 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:11.605259 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b2b88a-cce3-47b3-87a7-acaeb6434376 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:11.607153 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b2b88a-cce3-47b3-87a7-acaeb6434376 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:11.633447 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1250/2517] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:09:11 2022] POST /identity/v3/users => generated 295 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:09:11.634550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b2b88a-cce3-47b3-87a7-acaeb6434376 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:11.644156 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b2b88a-cce3-47b3-87a7-acaeb6434376 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:11.649067 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2e4e83-ec63-4500-92bc-21f7534581fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:11.649664 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2e4e83-ec63-4500-92bc-21f7534581fa admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:11.649938 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2e4e83-ec63-4500-92bc-21f7534581fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:11.650206 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2e4e83-ec63-4500-92bc-21f7534581fa admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:11.650978 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2e4e83-ec63-4500-92bc-21f7534581fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:11.652605 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2e4e83-ec63-4500-92bc-21f7534581fa admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:11.662988 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1251/2518] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:11 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:09:11.664033 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b688be5-d851-4684-9f54-dcf1aa42f5ef None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:11.667051 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2e4e83-ec63-4500-92bc-21f7534581fa admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:11.667682 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d676d37d-eb23-4cc6-b992-6fdbb584254f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:11.668403 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d676d37d-eb23-4cc6-b992-6fdbb584254f service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:11.668673 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d676d37d-eb23-4cc6-b992-6fdbb584254f service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:11.668937 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d676d37d-eb23-4cc6-b992-6fdbb584254f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:11.676990 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2e4e83-ec63-4500-92bc-21f7534581fa admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:11.678140 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d676d37d-eb23-4cc6-b992-6fdbb584254f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:11.679578 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d676d37d-eb23-4cc6-b992-6fdbb584254f service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:11.681892 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e509ce13-85b4-4bc5-ac58-a60adb20cef6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:11.682523 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e509ce13-85b4-4bc5-ac58-a60adb20cef6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:11.682907 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e509ce13-85b4-4bc5-ac58-a60adb20cef6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:11.683296 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e509ce13-85b4-4bc5-ac58-a60adb20cef6 admin admin] PATH_INFO: `/v3/projects/6536cb656297461cbf1fa8b352f5b815/users/20b3f20b192c421fb1e9df1f1fdb1835/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:11.689137 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1268/2519] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:09:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 01:09:11.701228 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e509ce13-85b4-4bc5-ac58-a60adb20cef6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6536cb656297461cbf1fa8b352f5b815, user_id=20b3f20b192c421fb1e9df1f1fdb1835, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:11.703943 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e509ce13-85b4-4bc5-ac58-a60adb20cef6 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:11.718374 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1252/2520] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:09:11 2022] PUT /identity/v3/projects/6536cb656297461cbf1fa8b352f5b815/users/20b3f20b192c421fb1e9df1f1fdb1835/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:11.722356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d676d37d-eb23-4cc6-b992-6fdbb584254f service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:11.732400 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d676d37d-eb23-4cc6-b992-6fdbb584254f service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:11.792426 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8698889b-f496-4461-98b9-d2da4334e5da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:11.793039 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8698889b-f496-4461-98b9-d2da4334e5da admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:11.793313 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8698889b-f496-4461-98b9-d2da4334e5da admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:11.793729 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8698889b-f496-4461-98b9-d2da4334e5da admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:11.794371 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8698889b-f496-4461-98b9-d2da4334e5da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:11.795990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8698889b-f496-4461-98b9-d2da4334e5da admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:11.809355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1269/2521] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:11 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 30 01:09:11.810529 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e509ce13-85b4-4bc5-ac58-a60adb20cef6 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:11.821265 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e509ce13-85b4-4bc5-ac58-a60adb20cef6 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:11.826067 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7424bae0-aac7-4144-973c-6029aedacb53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:11.826719 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7424bae0-aac7-4144-973c-6029aedacb53 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:11.826991 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7424bae0-aac7-4144-973c-6029aedacb53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:11.827257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7424bae0-aac7-4144-973c-6029aedacb53 admin admin] PATH_INFO: `/v3/projects/6536cb656297461cbf1fa8b352f5b815/users/20b3f20b192c421fb1e9df1f1fdb1835/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:11.829202 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7424bae0-aac7-4144-973c-6029aedacb53 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6536cb656297461cbf1fa8b352f5b815, user_id=20b3f20b192c421fb1e9df1f1fdb1835, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:11.831895 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7424bae0-aac7-4144-973c-6029aedacb53 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:11.851748 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1253/2522] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:09:11 2022] PUT /identity/v3/projects/6536cb656297461cbf1fa8b352f5b815/users/20b3f20b192c421fb1e9df1f1fdb1835/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:09:11.922176 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1201393a-c220-46a7-abbf-b9dd976445ef None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:11.922461 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1201393a-c220-46a7-abbf-b9dd976445ef None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:11.922719 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1201393a-c220-46a7-abbf-b9dd976445ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:11.951339 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-1201393a-c220-46a7-abbf-b9dd976445ef None None] MFA Rules not processed for user `20b3f20b192c421fb1e9df1f1fdb1835`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:11.952609 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-1201393a-c220-46a7-abbf-b9dd976445ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:11.976089 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7424bae0-aac7-4144-973c-6029aedacb53 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:11.978755 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-7424bae0-aac7-4144-973c-6029aedacb53 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:12.008480 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:12.034103 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1270/2523] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:11 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:12.045587 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1201393a-c220-46a7-abbf-b9dd976445ef None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:12.047251 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-1201393a-c220-46a7-abbf-b9dd976445ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:12.061370 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7424bae0-aac7-4144-973c-6029aedacb53 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:12.066055 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8931ef3-635b-48ad-be81-ce2ddfffe9c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:12.066680 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8931ef3-635b-48ad-be81-ce2ddfffe9c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:12.066953 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8931ef3-635b-48ad-be81-ce2ddfffe9c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:12.067218 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8931ef3-635b-48ad-be81-ce2ddfffe9c7 admin admin] PATH_INFO: `/v3/projects/183924fe30e949919847fc51afdec2d8` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:12.068464 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8931ef3-635b-48ad-be81-ce2ddfffe9c7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=183924fe30e949919847fc51afdec2d8)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:12.070714 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8931ef3-635b-48ad-be81-ce2ddfffe9c7 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:12.107362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1201393a-c220-46a7-abbf-b9dd976445ef None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:12.118250 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-d8931ef3-635b-48ad-be81-ce2ddfffe9c7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '183924fe30e949919847fc51afdec2d8', 'request_id': 'req-d8931ef3-635b-48ad-be81-ce2ddfffe9c7'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:12.120404 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=90,keystone:INSERT=5,keystone:DELETE=5 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:12.124470 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-d8931ef3-635b-48ad-be81-ce2ddfffe9c7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '183924fe30e949919847fc51afdec2d8', 'request_id': 'req-d8931ef3-635b-48ad-be81-ce2ddfffe9c7'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:12.133938 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-d8931ef3-635b-48ad-be81-ce2ddfffe9c7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '183924fe30e949919847fc51afdec2d8', 'request_id': 'req-d8931ef3-635b-48ad-be81-ce2ddfffe9c7'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:12.146165 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-d8931ef3-635b-48ad-be81-ce2ddfffe9c7 admin admin] The token cache is being invalidate because project 183924fe30e949919847fc51afdec2d8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:09:12.146447 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-d8931ef3-635b-48ad-be81-ce2ddfffe9c7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:12.147093 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-d8931ef3-635b-48ad-be81-ce2ddfffe9c7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:12.148854 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1254/2524] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:09:11 2022] DELETE /identity/v3/projects/183924fe30e949919847fc51afdec2d8 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:09:12.152952 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8931ef3-635b-48ad-be81-ce2ddfffe9c7 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:12.154308 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d8931ef3-635b-48ad-be81-ce2ddfffe9c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:12.156362 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5fb82cb-3ce7-4986-972d-b4b3f3e70620 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:12.156969 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fb82cb-3ce7-4986-972d-b4b3f3e70620 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:12.157242 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fb82cb-3ce7-4986-972d-b4b3f3e70620 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:12.157504 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fb82cb-3ce7-4986-972d-b4b3f3e70620 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:12.159161 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-a5fb82cb-3ce7-4986-972d-b4b3f3e70620 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:12.219085 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8931ef3-635b-48ad-be81-ce2ddfffe9c7 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:12.223683 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5fb82cb-3ce7-4986-972d-b4b3f3e70620 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:12.223858 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fabdab-56e8-45a8-924b-c2b4cef7e29e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:12.224388 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fabdab-56e8-45a8-924b-c2b4cef7e29e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:12.224661 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fabdab-56e8-45a8-924b-c2b4cef7e29e admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:12.224930 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fabdab-56e8-45a8-924b-c2b4cef7e29e admin admin] PATH_INFO: `/v3/users/a57b9efd3074463abe08e437fc3d49db` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:12.225090 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5fb82cb-3ce7-4986-972d-b4b3f3e70620 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:12.226078 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61fabdab-56e8-45a8-924b-c2b4cef7e29e admin admin] RBAC: Authorizing `identity:delete_user(user_id=a57b9efd3074463abe08e437fc3d49db)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:12.228019 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61fabdab-56e8-45a8-924b-c2b4cef7e29e admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:12.234799 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1271/2525] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:09:12 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 01:09:12.276816 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-61fabdab-56e8-45a8-924b-c2b4cef7e29e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a57b9efd3074463abe08e437fc3d49db'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:12.278519 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5fb82cb-3ce7-4986-972d-b4b3f3e70620 service cinder] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:12.280113 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-a5fb82cb-3ce7-4986-972d-b4b3f3e70620 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:12.286305 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-61fabdab-56e8-45a8-924b-c2b4cef7e29e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a57b9efd3074463abe08e437fc3d49db'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:12.286944 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-61fabdab-56e8-45a8-924b-c2b4cef7e29e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a57b9efd3074463abe08e437fc3d49db'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:12.291567 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5fb82cb-3ce7-4986-972d-b4b3f3e70620 service cinder] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:12.293980 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-61fabdab-56e8-45a8-924b-c2b4cef7e29e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a57b9efd3074463abe08e437fc3d49db'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:12.295325 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef328ec-d3a2-4994-b682-62a25f6b52f1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:12.295975 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef328ec-d3a2-4994-b682-62a25f6b52f1 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:12.296241 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef328ec-d3a2-4994-b682-62a25f6b52f1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:12.296503 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef328ec-d3a2-4994-b682-62a25f6b52f1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:12.298209 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3ef328ec-d3a2-4994-b682-62a25f6b52f1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:12.306539 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-61fabdab-56e8-45a8-924b-c2b4cef7e29e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a57b9efd3074463abe08e437fc3d49db'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:12.313367 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1255/2526] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:09:12 2022] DELETE /identity/v3/users/a57b9efd3074463abe08e437fc3d49db => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:09:12.349258 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=100 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:12.364067 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef328ec-d3a2-4994-b682-62a25f6b52f1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:12.365663 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef328ec-d3a2-4994-b682-62a25f6b52f1 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:12.375299 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1272/2527] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:09:12 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 01:09:12.605726 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fabdab-56e8-45a8-924b-c2b4cef7e29e admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:12.607708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-61fabdab-56e8-45a8-924b-c2b4cef7e29e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:12.692946 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fabdab-56e8-45a8-924b-c2b4cef7e29e admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:12.698000 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ea6856-d901-4a6d-934f-d089e87f04ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:12.698685 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ea6856-d901-4a6d-934f-d089e87f04ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:12.698911 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ea6856-d901-4a6d-934f-d089e87f04ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:12.699179 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ea6856-d901-4a6d-934f-d089e87f04ad admin admin] PATH_INFO: `/v3/projects/98cb66590aa1442fbc87e29a891159ed` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:12.700379 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ea6856-d901-4a6d-934f-d089e87f04ad admin admin] RBAC: Authorizing `identity:delete_project(project_id=98cb66590aa1442fbc87e29a891159ed)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:12.702375 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ea6856-d901-4a6d-934f-d089e87f04ad admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:12.705184 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:12.758174 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-75ea6856-d901-4a6d-934f-d089e87f04ad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '98cb66590aa1442fbc87e29a891159ed', 'request_id': 'req-75ea6856-d901-4a6d-934f-d089e87f04ad'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:12.764503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-75ea6856-d901-4a6d-934f-d089e87f04ad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '98cb66590aa1442fbc87e29a891159ed', 'request_id': 'req-75ea6856-d901-4a6d-934f-d089e87f04ad'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:12.777588 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-75ea6856-d901-4a6d-934f-d089e87f04ad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '98cb66590aa1442fbc87e29a891159ed', 'request_id': 'req-75ea6856-d901-4a6d-934f-d089e87f04ad'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:12.791362 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-75ea6856-d901-4a6d-934f-d089e87f04ad admin admin] The token cache is being invalidate because project 98cb66590aa1442fbc87e29a891159ed was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:09:12.791654 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-75ea6856-d901-4a6d-934f-d089e87f04ad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:12.792461 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-75ea6856-d901-4a6d-934f-d089e87f04ad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:12.793956 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1256/2528] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:09:12 2022] DELETE /identity/v3/projects/98cb66590aa1442fbc87e29a891159ed => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:15.355731 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef328ec-d3a2-4994-b682-62a25f6b52f1 service cinder] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:15.357237 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3ef328ec-d3a2-4994-b682-62a25f6b52f1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:15.371589 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef328ec-d3a2-4994-b682-62a25f6b52f1 service cinder] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:15.429735 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b05aa5-38ce-4d03-9148-e9fa94a944e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:15.430368 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b05aa5-38ce-4d03-9148-e9fa94a944e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:15.430637 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b05aa5-38ce-4d03-9148-e9fa94a944e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:15.430901 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b05aa5-38ce-4d03-9148-e9fa94a944e8 admin admin] PATH_INFO: `/v3/users/f45fa00e4e6144f79724f9d542cf3c87` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:15.432108 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b05aa5-38ce-4d03-9148-e9fa94a944e8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f45fa00e4e6144f79724f9d542cf3c87)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:15.434032 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b05aa5-38ce-4d03-9148-e9fa94a944e8 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:15.483391 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-66b05aa5-38ce-4d03-9148-e9fa94a944e8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f45fa00e4e6144f79724f9d542cf3c87'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:15.492492 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-66b05aa5-38ce-4d03-9148-e9fa94a944e8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f45fa00e4e6144f79724f9d542cf3c87'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:15.493187 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-66b05aa5-38ce-4d03-9148-e9fa94a944e8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f45fa00e4e6144f79724f9d542cf3c87'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:15.506639 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-66b05aa5-38ce-4d03-9148-e9fa94a944e8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f45fa00e4e6144f79724f9d542cf3c87'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:15.513746 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-66b05aa5-38ce-4d03-9148-e9fa94a944e8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f45fa00e4e6144f79724f9d542cf3c87'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:15.520236 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1273/2529] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:09:15 2022] DELETE /identity/v3/users/f45fa00e4e6144f79724f9d542cf3c87 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:15.525071 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa9654e-a816-4498-b95c-ce2b3cc34c08 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:15.525593 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa9654e-a816-4498-b95c-ce2b3cc34c08 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:15.525593 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa9654e-a816-4498-b95c-ce2b3cc34c08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:15.546015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-2fa9654e-a816-4498-b95c-ce2b3cc34c08 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:15.547213 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-2fa9654e-a816-4498-b95c-ce2b3cc34c08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:15.616895 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1257/2530] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:15.627042 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b05aa5-38ce-4d03-9148-e9fa94a944e8 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:15.628746 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-66b05aa5-38ce-4d03-9148-e9fa94a944e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:15.688062 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b05aa5-38ce-4d03-9148-e9fa94a944e8 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:15.692187 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73120a7b-bf1d-46a7-8e1c-bd149d5fb0a6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:15.692790 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73120a7b-bf1d-46a7-8e1c-bd149d5fb0a6 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:15.693065 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73120a7b-bf1d-46a7-8e1c-bd149d5fb0a6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:15.693332 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73120a7b-bf1d-46a7-8e1c-bd149d5fb0a6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:15.702553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73120a7b-bf1d-46a7-8e1c-bd149d5fb0a6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:15.704055 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73120a7b-bf1d-46a7-8e1c-bd149d5fb0a6 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:15.713850 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1274/2531] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:09:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:09:15.998981 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa9654e-a816-4498-b95c-ce2b3cc34c08 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:16.000831 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-2fa9654e-a816-4498-b95c-ce2b3cc34c08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:16.013590 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa9654e-a816-4498-b95c-ce2b3cc34c08 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:16.018357 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818d548d-b838-4cc5-83aa-b5260de9f38f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:16.019848 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818d548d-b838-4cc5-83aa-b5260de9f38f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:16.020120 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818d548d-b838-4cc5-83aa-b5260de9f38f admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:16.020383 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818d548d-b838-4cc5-83aa-b5260de9f38f admin admin] PATH_INFO: `/v3/projects/6c7a42e7e9ae438da36e6ac6ed9d4712` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:16.021859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818d548d-b838-4cc5-83aa-b5260de9f38f admin admin] RBAC: Authorizing `identity:delete_project(project_id=6c7a42e7e9ae438da36e6ac6ed9d4712)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:16.023755 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818d548d-b838-4cc5-83aa-b5260de9f38f admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:16.075271 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-818d548d-b838-4cc5-83aa-b5260de9f38f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6c7a42e7e9ae438da36e6ac6ed9d4712', 'request_id': 'req-818d548d-b838-4cc5-83aa-b5260de9f38f'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:16.080573 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-818d548d-b838-4cc5-83aa-b5260de9f38f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6c7a42e7e9ae438da36e6ac6ed9d4712', 'request_id': 'req-818d548d-b838-4cc5-83aa-b5260de9f38f'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:16.089204 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-818d548d-b838-4cc5-83aa-b5260de9f38f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6c7a42e7e9ae438da36e6ac6ed9d4712', 'request_id': 'req-818d548d-b838-4cc5-83aa-b5260de9f38f'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:16.101201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-818d548d-b838-4cc5-83aa-b5260de9f38f admin admin] The token cache is being invalidate because project 6c7a42e7e9ae438da36e6ac6ed9d4712 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:09:16.101486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-818d548d-b838-4cc5-83aa-b5260de9f38f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:16.102111 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-818d548d-b838-4cc5-83aa-b5260de9f38f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:16.103648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1258/2532] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:09:15 2022] DELETE /identity/v3/projects/6c7a42e7e9ae438da36e6ac6ed9d4712 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:16.107874 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73120a7b-bf1d-46a7-8e1c-bd149d5fb0a6 service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:16.118156 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73120a7b-bf1d-46a7-8e1c-bd149d5fb0a6 service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:16.122894 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:16.177068 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d106a2fb-9f96-4196-abf2-e5b2d91abe27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:16.177698 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d106a2fb-9f96-4196-abf2-e5b2d91abe27 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:16.177975 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d106a2fb-9f96-4196-abf2-e5b2d91abe27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:16.178232 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d106a2fb-9f96-4196-abf2-e5b2d91abe27 admin admin] PATH_INFO: `/v3/users/9e9a69578c6546b99a67073a2a033452` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:16.179353 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d106a2fb-9f96-4196-abf2-e5b2d91abe27 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9e9a69578c6546b99a67073a2a033452)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:16.181316 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d106a2fb-9f96-4196-abf2-e5b2d91abe27 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:16.230227 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-d106a2fb-9f96-4196-abf2-e5b2d91abe27 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9e9a69578c6546b99a67073a2a033452'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:16.238998 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-d106a2fb-9f96-4196-abf2-e5b2d91abe27 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9e9a69578c6546b99a67073a2a033452'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:16.239590 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-d106a2fb-9f96-4196-abf2-e5b2d91abe27 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9e9a69578c6546b99a67073a2a033452'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:16.251543 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-d106a2fb-9f96-4196-abf2-e5b2d91abe27 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9e9a69578c6546b99a67073a2a033452'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:16.259036 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-d106a2fb-9f96-4196-abf2-e5b2d91abe27 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9e9a69578c6546b99a67073a2a033452'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:16.265441 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1275/2533] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:09:16 2022] DELETE /identity/v3/users/9e9a69578c6546b99a67073a2a033452 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:16.548995 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818d548d-b838-4cc5-83aa-b5260de9f38f admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:16.550568 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-818d548d-b838-4cc5-83aa-b5260de9f38f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:16.563861 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818d548d-b838-4cc5-83aa-b5260de9f38f admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:16.573101 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:16.623227 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b93159e-7577-4e22-acec-db72a39409ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:16.623819 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b93159e-7577-4e22-acec-db72a39409ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:16.624098 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b93159e-7577-4e22-acec-db72a39409ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:16.624368 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b93159e-7577-4e22-acec-db72a39409ca admin admin] PATH_INFO: `/v3/projects/541e27b1868841b8b4ba09030ea67bfe` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:16.625501 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b93159e-7577-4e22-acec-db72a39409ca admin admin] RBAC: Authorizing `identity:delete_project(project_id=541e27b1868841b8b4ba09030ea67bfe)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:16.627398 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b93159e-7577-4e22-acec-db72a39409ca admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:16.687687 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-3b93159e-7577-4e22-acec-db72a39409ca admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '541e27b1868841b8b4ba09030ea67bfe', 'request_id': 'req-3b93159e-7577-4e22-acec-db72a39409ca'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:16.693660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-3b93159e-7577-4e22-acec-db72a39409ca admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '541e27b1868841b8b4ba09030ea67bfe', 'request_id': 'req-3b93159e-7577-4e22-acec-db72a39409ca'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:16.702819 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-3b93159e-7577-4e22-acec-db72a39409ca admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '541e27b1868841b8b4ba09030ea67bfe', 'request_id': 'req-3b93159e-7577-4e22-acec-db72a39409ca'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:16.715088 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-3b93159e-7577-4e22-acec-db72a39409ca admin admin] The token cache is being invalidate because project 541e27b1868841b8b4ba09030ea67bfe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:09:16.715368 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-3b93159e-7577-4e22-acec-db72a39409ca admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:16.716024 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-3b93159e-7577-4e22-acec-db72a39409ca admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:16.717513 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1259/2534] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:09:16 2022] DELETE /identity/v3/projects/541e27b1868841b8b4ba09030ea67bfe => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:16.721941 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d106a2fb-9f96-4196-abf2-e5b2d91abe27 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:16.723537 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-d106a2fb-9f96-4196-abf2-e5b2d91abe27 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:16.736392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d106a2fb-9f96-4196-abf2-e5b2d91abe27 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:16.796009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2397304-2644-415b-bcfb-a42e9dd7997f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:16.796617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2397304-2644-415b-bcfb-a42e9dd7997f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:16.796914 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2397304-2644-415b-bcfb-a42e9dd7997f admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:16.797217 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2397304-2644-415b-bcfb-a42e9dd7997f admin admin] PATH_INFO: `/v3/users/20b3f20b192c421fb1e9df1f1fdb1835` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:16.798361 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2397304-2644-415b-bcfb-a42e9dd7997f admin admin] RBAC: Authorizing `identity:delete_user(user_id=20b3f20b192c421fb1e9df1f1fdb1835)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:16.800321 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2397304-2644-415b-bcfb-a42e9dd7997f admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:16.836238 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:16.851045 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a2397304-2644-415b-bcfb-a42e9dd7997f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '20b3f20b192c421fb1e9df1f1fdb1835'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:16.859623 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a2397304-2644-415b-bcfb-a42e9dd7997f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '20b3f20b192c421fb1e9df1f1fdb1835'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:16.860256 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a2397304-2644-415b-bcfb-a42e9dd7997f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '20b3f20b192c421fb1e9df1f1fdb1835'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:16.872163 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a2397304-2644-415b-bcfb-a42e9dd7997f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '20b3f20b192c421fb1e9df1f1fdb1835'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:16.879129 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a2397304-2644-415b-bcfb-a42e9dd7997f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '20b3f20b192c421fb1e9df1f1fdb1835'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:16.885447 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1276/2535] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:09:16 2022] DELETE /identity/v3/users/20b3f20b192c421fb1e9df1f1fdb1835 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:17.178893 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b93159e-7577-4e22-acec-db72a39409ca admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:17.180667 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-3b93159e-7577-4e22-acec-db72a39409ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:17.195057 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b93159e-7577-4e22-acec-db72a39409ca admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:17.255504 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9166b14-c757-4899-b9ec-a6e2590ef2e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:17.256154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9166b14-c757-4899-b9ec-a6e2590ef2e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:17.256427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9166b14-c757-4899-b9ec-a6e2590ef2e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:17.256694 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9166b14-c757-4899-b9ec-a6e2590ef2e4 admin admin] PATH_INFO: `/v3/projects/6536cb656297461cbf1fa8b352f5b815` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:17.257853 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9166b14-c757-4899-b9ec-a6e2590ef2e4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6536cb656297461cbf1fa8b352f5b815)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:17.259857 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9166b14-c757-4899-b9ec-a6e2590ef2e4 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:17.316833 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:17.319708 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b9166b14-c757-4899-b9ec-a6e2590ef2e4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6536cb656297461cbf1fa8b352f5b815', 'request_id': 'req-b9166b14-c757-4899-b9ec-a6e2590ef2e4'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:17.327003 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b9166b14-c757-4899-b9ec-a6e2590ef2e4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6536cb656297461cbf1fa8b352f5b815', 'request_id': 'req-b9166b14-c757-4899-b9ec-a6e2590ef2e4'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:17.336126 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b9166b14-c757-4899-b9ec-a6e2590ef2e4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6536cb656297461cbf1fa8b352f5b815', 'request_id': 'req-b9166b14-c757-4899-b9ec-a6e2590ef2e4'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:17.349700 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b9166b14-c757-4899-b9ec-a6e2590ef2e4 admin admin] The token cache is being invalidate because project 6536cb656297461cbf1fa8b352f5b815 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:09:17.349985 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b9166b14-c757-4899-b9ec-a6e2590ef2e4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:17.350664 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b9166b14-c757-4899-b9ec-a6e2590ef2e4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:17.352291 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1260/2536] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:09:17 2022] DELETE /identity/v3/projects/6536cb656297461cbf1fa8b352f5b815 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 30 01:09:17.358497 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c82be01-fed4-4897-bb47-fd2e514686b7 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:17.358780 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c82be01-fed4-4897-bb47-fd2e514686b7 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:17.359045 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c82be01-fed4-4897-bb47-fd2e514686b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:17.379966 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-1c82be01-fed4-4897-bb47-fd2e514686b7 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:17.381156 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-1c82be01-fed4-4897-bb47-fd2e514686b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:17.442549 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1277/2537] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:17.451295 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73cdda6-b6a6-4238-95e4-181f5ee9ce17 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:17.451579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73cdda6-b6a6-4238-95e4-181f5ee9ce17 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:17.451868 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73cdda6-b6a6-4238-95e4-181f5ee9ce17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:17.473552 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-f73cdda6-b6a6-4238-95e4-181f5ee9ce17 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:17.474742 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-f73cdda6-b6a6-4238-95e4-181f5ee9ce17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:17.483761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1261/2538] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:17.488096 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c82be01-fed4-4897-bb47-fd2e514686b7 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:17.498004 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c82be01-fed4-4897-bb47-fd2e514686b7 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:17.503086 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239c5c9e-8b16-4498-8ad0-0c25d573837d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:17.503668 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239c5c9e-8b16-4498-8ad0-0c25d573837d admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:17.503966 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239c5c9e-8b16-4498-8ad0-0c25d573837d admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:17.504284 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239c5c9e-8b16-4498-8ad0-0c25d573837d admin admin] PATH_INFO: `/v3/domains` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:17.505009 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239c5c9e-8b16-4498-8ad0-0c25d573837d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81495) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:09:17.505300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239c5c9e-8b16-4498-8ad0-0c25d573837d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:17.506868 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239c5c9e-8b16-4498-8ad0-0c25d573837d admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:17.518877 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1278/2539] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:09:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 01:09:17.523483 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73cdda6-b6a6-4238-95e4-181f5ee9ce17 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:17.532833 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73cdda6-b6a6-4238-95e4-181f5ee9ce17 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:17.538262 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1c2499-e11c-4028-b34c-2f9168fcc0d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:17.538890 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1c2499-e11c-4028-b34c-2f9168fcc0d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:17.539156 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1c2499-e11c-4028-b34c-2f9168fcc0d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:17.539426 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1c2499-e11c-4028-b34c-2f9168fcc0d4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:17.540067 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1c2499-e11c-4028-b34c-2f9168fcc0d4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:17.541993 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1c2499-e11c-4028-b34c-2f9168fcc0d4 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:17.574263 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1262/2540] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:09:17 2022] POST /identity/v3/projects => generated 379 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:09:17.578474 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239c5c9e-8b16-4498-8ad0-0c25d573837d admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:17.589012 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239c5c9e-8b16-4498-8ad0-0c25d573837d admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:17.647561 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df364835-4480-4cde-acb2-a121c1106b43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:17.648204 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df364835-4480-4cde-acb2-a121c1106b43 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:17.648479 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df364835-4480-4cde-acb2-a121c1106b43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:17.648740 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df364835-4480-4cde-acb2-a121c1106b43 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:17.649364 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df364835-4480-4cde-acb2-a121c1106b43 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:17.651272 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df364835-4480-4cde-acb2-a121c1106b43 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:17.675955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1279/2541] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:09:17 2022] POST /identity/v3/users => generated 290 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:09:17.680314 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1c2499-e11c-4028-b34c-2f9168fcc0d4 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:17.690847 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1c2499-e11c-4028-b34c-2f9168fcc0d4 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:17.696213 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b45bc84-08e3-43e7-9adc-b548830de6cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:17.696727 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b45bc84-08e3-43e7-9adc-b548830de6cd admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:17.696999 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b45bc84-08e3-43e7-9adc-b548830de6cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:17.697263 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b45bc84-08e3-43e7-9adc-b548830de6cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:17.697856 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b45bc84-08e3-43e7-9adc-b548830de6cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:17.699475 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b45bc84-08e3-43e7-9adc-b548830de6cd admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:17.710448 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1263/2542] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:17 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:09:17.714634 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df364835-4480-4cde-acb2-a121c1106b43 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:17.724990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df364835-4480-4cde-acb2-a121c1106b43 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:17.730675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56358b2f-8eb4-4f19-bc5b-17383eb0ed2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:17.731268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56358b2f-8eb4-4f19-bc5b-17383eb0ed2d admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:17.731543 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56358b2f-8eb4-4f19-bc5b-17383eb0ed2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:17.731856 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56358b2f-8eb4-4f19-bc5b-17383eb0ed2d admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:17.732478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56358b2f-8eb4-4f19-bc5b-17383eb0ed2d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:17.734126 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56358b2f-8eb4-4f19-bc5b-17383eb0ed2d admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:17.745142 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1280/2543] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:17 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:09:17.749358 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b45bc84-08e3-43e7-9adc-b548830de6cd admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:17.759439 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b45bc84-08e3-43e7-9adc-b548830de6cd admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:17.764602 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a9841c-da08-428c-b897-fe737e654d0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:17.765236 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a9841c-da08-428c-b897-fe737e654d0a admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:17.765503 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a9841c-da08-428c-b897-fe737e654d0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:17.765764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a9841c-da08-428c-b897-fe737e654d0a admin admin] PATH_INFO: `/v3/projects/1c76326d37fb41948a53e64cc66b5592/users/f876d0d20e7a49b98c47467db4758a82/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:17.780903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a9841c-da08-428c-b897-fe737e654d0a admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c76326d37fb41948a53e64cc66b5592, user_id=f876d0d20e7a49b98c47467db4758a82, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:17.783516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a9841c-da08-428c-b897-fe737e654d0a admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:17.798813 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1264/2544] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:09:17 2022] PUT /identity/v3/projects/1c76326d37fb41948a53e64cc66b5592/users/f876d0d20e7a49b98c47467db4758a82/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:17.810289 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b4c714-41e8-4bbc-a9d1-f5894e02f235 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:17.810568 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b4c714-41e8-4bbc-a9d1-f5894e02f235 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:17.810864 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b4c714-41e8-4bbc-a9d1-f5894e02f235 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:17.836801 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:17.840593 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-c4b4c714-41e8-4bbc-a9d1-f5894e02f235 None None] MFA Rules not processed for user `f876d0d20e7a49b98c47467db4758a82`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:17.843027 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c4b4c714-41e8-4bbc-a9d1-f5894e02f235 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:17.914481 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1281/2545] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:17 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:17.919349 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a9841c-da08-428c-b897-fe737e654d0a admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:17.993276 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a9841c-da08-428c-b897-fe737e654d0a admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:17.999706 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5324f4b7-7ec2-4f82-b230-48c711351fbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:18.000344 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5324f4b7-7ec2-4f82-b230-48c711351fbb admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:18.000625 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5324f4b7-7ec2-4f82-b230-48c711351fbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:18.000879 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5324f4b7-7ec2-4f82-b230-48c711351fbb admin admin] PATH_INFO: `/v3/projects` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:18.001505 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5324f4b7-7ec2-4f82-b230-48c711351fbb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:18.003458 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5324f4b7-7ec2-4f82-b230-48c711351fbb admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:18.034158 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1265/2546] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:09:17 2022] POST /identity/v3/projects => generated 381 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:09:18.038554 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b4c714-41e8-4bbc-a9d1-f5894e02f235 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:18.050300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b4c714-41e8-4bbc-a9d1-f5894e02f235 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:18.114597 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d923754f-6f23-4c3c-bc13-3d79bf11f2a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:18.115195 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d923754f-6f23-4c3c-bc13-3d79bf11f2a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:18.115467 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d923754f-6f23-4c3c-bc13-3d79bf11f2a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:18.115757 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d923754f-6f23-4c3c-bc13-3d79bf11f2a2 admin admin] PATH_INFO: `/v3/users` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:18.116363 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d923754f-6f23-4c3c-bc13-3d79bf11f2a2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:18.118259 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d923754f-6f23-4c3c-bc13-3d79bf11f2a2 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:18.144270 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1282/2547] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:09:18 2022] POST /identity/v3/users => generated 297 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:09:18.145598 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5324f4b7-7ec2-4f82-b230-48c711351fbb admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:18.156149 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5324f4b7-7ec2-4f82-b230-48c711351fbb admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:18.163001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d34811e-0366-4c4c-85a2-50e8d25e7e27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:18.163596 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d34811e-0366-4c4c-85a2-50e8d25e7e27 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:18.163907 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d34811e-0366-4c4c-85a2-50e8d25e7e27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:18.164175 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d34811e-0366-4c4c-85a2-50e8d25e7e27 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:18.164835 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d34811e-0366-4c4c-85a2-50e8d25e7e27 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:18.166441 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d34811e-0366-4c4c-85a2-50e8d25e7e27 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:18.177574 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1266/2548] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:18 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:09:18.183749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d923754f-6f23-4c3c-bc13-3d79bf11f2a2 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:18.194439 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d923754f-6f23-4c3c-bc13-3d79bf11f2a2 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:18.199469 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6415aeff-e8aa-4546-b85c-1a996f355f5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:18.200146 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6415aeff-e8aa-4546-b85c-1a996f355f5a admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:18.200422 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6415aeff-e8aa-4546-b85c-1a996f355f5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:18.200693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6415aeff-e8aa-4546-b85c-1a996f355f5a admin admin] PATH_INFO: `/v3/projects/d986f0c89b9b45a8a074d52b360a1fdb/users/13569144620049a2b9d7b9491e2f8864/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:18.216777 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6415aeff-e8aa-4546-b85c-1a996f355f5a admin admin] RBAC: Authorizing `identity:create_grant(project_id=d986f0c89b9b45a8a074d52b360a1fdb, user_id=13569144620049a2b9d7b9491e2f8864, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:18.219436 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6415aeff-e8aa-4546-b85c-1a996f355f5a admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:18.235295 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1283/2549] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:09:18 2022] PUT /identity/v3/projects/d986f0c89b9b45a8a074d52b360a1fdb/users/13569144620049a2b9d7b9491e2f8864/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:18.239908 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d34811e-0366-4c4c-85a2-50e8d25e7e27 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:18.250030 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d34811e-0366-4c4c-85a2-50e8d25e7e27 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:18.292153 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=4 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:18.309456 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dad3686-0e5f-4854-bff7-88db3a32ef01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:18.310066 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dad3686-0e5f-4854-bff7-88db3a32ef01 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:18.310341 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dad3686-0e5f-4854-bff7-88db3a32ef01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:18.310613 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dad3686-0e5f-4854-bff7-88db3a32ef01 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:18.311311 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dad3686-0e5f-4854-bff7-88db3a32ef01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:18.312942 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dad3686-0e5f-4854-bff7-88db3a32ef01 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:18.323128 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1267/2550] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:18 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:09:18.327752 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6415aeff-e8aa-4546-b85c-1a996f355f5a admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:18.339235 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6415aeff-e8aa-4546-b85c-1a996f355f5a admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:18.344868 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc84d523-9ee4-4b90-ac1a-e4a0b37b4faf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:18.346098 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc84d523-9ee4-4b90-ac1a-e4a0b37b4faf admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:18.346565 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc84d523-9ee4-4b90-ac1a-e4a0b37b4faf admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:18.347016 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc84d523-9ee4-4b90-ac1a-e4a0b37b4faf admin admin] PATH_INFO: `/v3/projects/d986f0c89b9b45a8a074d52b360a1fdb/users/13569144620049a2b9d7b9491e2f8864/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:18.349346 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc84d523-9ee4-4b90-ac1a-e4a0b37b4faf admin admin] RBAC: Authorizing `identity:create_grant(project_id=d986f0c89b9b45a8a074d52b360a1fdb, user_id=13569144620049a2b9d7b9491e2f8864, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:18.352017 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc84d523-9ee4-4b90-ac1a-e4a0b37b4faf admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:18.363644 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1284/2551] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:09:18 2022] PUT /identity/v3/projects/d986f0c89b9b45a8a074d52b360a1fdb/users/13569144620049a2b9d7b9491e2f8864/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:18.374235 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9f8649-8187-4512-87c8-620c7c12551d None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:18.374516 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9f8649-8187-4512-87c8-620c7c12551d None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:18.374785 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9f8649-8187-4512-87c8-620c7c12551d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:18.405933 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-cc9f8649-8187-4512-87c8-620c7c12551d None None] MFA Rules not processed for user `13569144620049a2b9d7b9491e2f8864`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:18.407178 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-cc9f8649-8187-4512-87c8-620c7c12551d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:18.477515 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1268/2552] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:18 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 30 01:09:18.489281 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc84d523-9ee4-4b90-ac1a-e4a0b37b4faf admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:18.491320 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-bc84d523-9ee4-4b90-ac1a-e4a0b37b4faf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:18.552913 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc84d523-9ee4-4b90-ac1a-e4a0b37b4faf admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:18.556711 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8c67c6-75c2-4490-8a48-150b3d90a033 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:18.557319 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8c67c6-75c2-4490-8a48-150b3d90a033 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:18.557594 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8c67c6-75c2-4490-8a48-150b3d90a033 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:18.557859 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8c67c6-75c2-4490-8a48-150b3d90a033 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:18.566978 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8c67c6-75c2-4490-8a48-150b3d90a033 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:18.568528 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8c67c6-75c2-4490-8a48-150b3d90a033 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:18.572337 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:18.580029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1285/2553] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:09:18 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:09:18.970154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9f8649-8187-4512-87c8-620c7c12551d None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:19.036337 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9f8649-8187-4512-87c8-620c7c12551d None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:19.041772 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d29622-eade-4082-8b51-e9f72eaf67ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:19.042373 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d29622-eade-4082-8b51-e9f72eaf67ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:19.042665 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d29622-eade-4082-8b51-e9f72eaf67ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:19.042932 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d29622-eade-4082-8b51-e9f72eaf67ca admin admin] PATH_INFO: `/v3/users/f876d0d20e7a49b98c47467db4758a82` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:19.044201 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d29622-eade-4082-8b51-e9f72eaf67ca admin admin] RBAC: Authorizing `identity:delete_user(user_id=f876d0d20e7a49b98c47467db4758a82)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:19.046112 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d29622-eade-4082-8b51-e9f72eaf67ca admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:19.096844 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-26d29622-eade-4082-8b51-e9f72eaf67ca admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f876d0d20e7a49b98c47467db4758a82'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:19.105866 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-26d29622-eade-4082-8b51-e9f72eaf67ca admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f876d0d20e7a49b98c47467db4758a82'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:19.106484 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-26d29622-eade-4082-8b51-e9f72eaf67ca admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f876d0d20e7a49b98c47467db4758a82'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:19.114018 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-26d29622-eade-4082-8b51-e9f72eaf67ca admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f876d0d20e7a49b98c47467db4758a82'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:19.118487 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:19.127541 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-26d29622-eade-4082-8b51-e9f72eaf67ca admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f876d0d20e7a49b98c47467db4758a82'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:19.133800 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1269/2554] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:09:18 2022] DELETE /identity/v3/users/f876d0d20e7a49b98c47467db4758a82 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:19.138414 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4175c5ed-d5a3-49f9-899d-e6cad723eaa8 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:19.138907 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4175c5ed-d5a3-49f9-899d-e6cad723eaa8 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:19.138907 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4175c5ed-d5a3-49f9-899d-e6cad723eaa8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:19.158516 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-4175c5ed-d5a3-49f9-899d-e6cad723eaa8 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:19.159738 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-4175c5ed-d5a3-49f9-899d-e6cad723eaa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:19.222929 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1286/2555] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:19.234455 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d29622-eade-4082-8b51-e9f72eaf67ca admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:19.236050 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-26d29622-eade-4082-8b51-e9f72eaf67ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:19.295650 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d29622-eade-4082-8b51-e9f72eaf67ca admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:19.300116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7e25a3-669f-4efb-926e-f8c0f26d28db service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:19.300724 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7e25a3-669f-4efb-926e-f8c0f26d28db service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:19.301073 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7e25a3-669f-4efb-926e-f8c0f26d28db service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:19.301353 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7e25a3-669f-4efb-926e-f8c0f26d28db service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:19.311275 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7e25a3-669f-4efb-926e-f8c0f26d28db service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:19.312747 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7e25a3-669f-4efb-926e-f8c0f26d28db service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:19.322711 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1270/2556] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:09:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:09:19.605056 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4175c5ed-d5a3-49f9-899d-e6cad723eaa8 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:19.606666 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-4175c5ed-d5a3-49f9-899d-e6cad723eaa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:19.624019 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4175c5ed-d5a3-49f9-899d-e6cad723eaa8 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:19.630907 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47e0dad-dbe3-4275-a987-99cd328bead7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:19.631827 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47e0dad-dbe3-4275-a987-99cd328bead7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:19.632080 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47e0dad-dbe3-4275-a987-99cd328bead7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:19.632328 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47e0dad-dbe3-4275-a987-99cd328bead7 admin admin] PATH_INFO: `/v3/projects/1c76326d37fb41948a53e64cc66b5592` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:19.633759 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47e0dad-dbe3-4275-a987-99cd328bead7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1c76326d37fb41948a53e64cc66b5592)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:19.635689 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47e0dad-dbe3-4275-a987-99cd328bead7 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:19.686357 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c47e0dad-dbe3-4275-a987-99cd328bead7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1c76326d37fb41948a53e64cc66b5592', 'request_id': 'req-c47e0dad-dbe3-4275-a987-99cd328bead7'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:19.694485 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c47e0dad-dbe3-4275-a987-99cd328bead7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1c76326d37fb41948a53e64cc66b5592', 'request_id': 'req-c47e0dad-dbe3-4275-a987-99cd328bead7'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:19.700057 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c47e0dad-dbe3-4275-a987-99cd328bead7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1c76326d37fb41948a53e64cc66b5592', 'request_id': 'req-c47e0dad-dbe3-4275-a987-99cd328bead7'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:19.711831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c47e0dad-dbe3-4275-a987-99cd328bead7 admin admin] The token cache is being invalidate because project 1c76326d37fb41948a53e64cc66b5592 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:09:19.712113 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c47e0dad-dbe3-4275-a987-99cd328bead7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:19.712734 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-c47e0dad-dbe3-4275-a987-99cd328bead7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:19.714216 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1287/2557] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:09:19 2022] DELETE /identity/v3/projects/1c76326d37fb41948a53e64cc66b5592 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:19.718423 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7e25a3-669f-4efb-926e-f8c0f26d28db service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:19.728786 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7e25a3-669f-4efb-926e-f8c0f26d28db service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:19.789613 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d7c66f-6585-4727-9b42-b34c054f8f38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:19.790559 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d7c66f-6585-4727-9b42-b34c054f8f38 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:19.790939 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d7c66f-6585-4727-9b42-b34c054f8f38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:19.791506 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d7c66f-6585-4727-9b42-b34c054f8f38 admin admin] PATH_INFO: `/v3/users/13569144620049a2b9d7b9491e2f8864` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:19.793950 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d7c66f-6585-4727-9b42-b34c054f8f38 admin admin] RBAC: Authorizing `identity:delete_user(user_id=13569144620049a2b9d7b9491e2f8864)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:19.796910 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d7c66f-6585-4727-9b42-b34c054f8f38 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:19.850279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-e4d7c66f-6585-4727-9b42-b34c054f8f38 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '13569144620049a2b9d7b9491e2f8864'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:19.859660 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-e4d7c66f-6585-4727-9b42-b34c054f8f38 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '13569144620049a2b9d7b9491e2f8864'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:19.860364 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-e4d7c66f-6585-4727-9b42-b34c054f8f38 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '13569144620049a2b9d7b9491e2f8864'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:19.867411 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-e4d7c66f-6585-4727-9b42-b34c054f8f38 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '13569144620049a2b9d7b9491e2f8864'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:19.879084 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-e4d7c66f-6585-4727-9b42-b34c054f8f38 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '13569144620049a2b9d7b9491e2f8864'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:19.885170 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1271/2558] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:09:19 2022] DELETE /identity/v3/users/13569144620049a2b9d7b9491e2f8864 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:20.170529 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47e0dad-dbe3-4275-a987-99cd328bead7 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:20.172042 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c47e0dad-dbe3-4275-a987-99cd328bead7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:20.185643 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47e0dad-dbe3-4275-a987-99cd328bead7 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:20.246721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745a8007-2c91-42ec-bceb-6882931a7b82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:20.247356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745a8007-2c91-42ec-bceb-6882931a7b82 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:20.247627 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745a8007-2c91-42ec-bceb-6882931a7b82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:20.247948 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745a8007-2c91-42ec-bceb-6882931a7b82 admin admin] PATH_INFO: `/v3/projects/d986f0c89b9b45a8a074d52b360a1fdb` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:20.249071 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745a8007-2c91-42ec-bceb-6882931a7b82 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d986f0c89b9b45a8a074d52b360a1fdb)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:20.250964 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745a8007-2c91-42ec-bceb-6882931a7b82 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:20.274905 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:20.301388 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-745a8007-2c91-42ec-bceb-6882931a7b82 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd986f0c89b9b45a8a074d52b360a1fdb', 'request_id': 'req-745a8007-2c91-42ec-bceb-6882931a7b82'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:20.311420 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-745a8007-2c91-42ec-bceb-6882931a7b82 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd986f0c89b9b45a8a074d52b360a1fdb', 'request_id': 'req-745a8007-2c91-42ec-bceb-6882931a7b82'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:20.317151 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-745a8007-2c91-42ec-bceb-6882931a7b82 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd986f0c89b9b45a8a074d52b360a1fdb', 'request_id': 'req-745a8007-2c91-42ec-bceb-6882931a7b82'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:20.328423 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-745a8007-2c91-42ec-bceb-6882931a7b82 admin admin] The token cache is being invalidate because project d986f0c89b9b45a8a074d52b360a1fdb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:09:20.328721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-745a8007-2c91-42ec-bceb-6882931a7b82 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:20.329351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-745a8007-2c91-42ec-bceb-6882931a7b82 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:20.330815 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1288/2559] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:09:20 2022] DELETE /identity/v3/projects/d986f0c89b9b45a8a074d52b360a1fdb => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:20.337359 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493c11cc-789b-49f1-864d-29f48daeface None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:20.337859 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493c11cc-789b-49f1-864d-29f48daeface None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:20.337981 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493c11cc-789b-49f1-864d-29f48daeface None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:20.353236 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:20.359942 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-493c11cc-789b-49f1-864d-29f48daeface None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:20.361564 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-493c11cc-789b-49f1-864d-29f48daeface None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:20.420938 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1272/2560] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:20.429832 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b57d42-7caa-4d60-ae22-0e136b1dd79a None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:20.430183 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b57d42-7caa-4d60-ae22-0e136b1dd79a None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:20.430553 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b57d42-7caa-4d60-ae22-0e136b1dd79a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:20.454592 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-38b57d42-7caa-4d60-ae22-0e136b1dd79a None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:20.455818 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-38b57d42-7caa-4d60-ae22-0e136b1dd79a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:20.465114 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1289/2561] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:20.469707 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493c11cc-789b-49f1-864d-29f48daeface None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:20.480695 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493c11cc-789b-49f1-864d-29f48daeface None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:20.486841 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7e25f8-8e38-456e-bcf1-18a6900bc3f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:20.487430 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7e25f8-8e38-456e-bcf1-18a6900bc3f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:20.487699 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7e25f8-8e38-456e-bcf1-18a6900bc3f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:20.487981 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7e25f8-8e38-456e-bcf1-18a6900bc3f4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:20.488706 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7e25f8-8e38-456e-bcf1-18a6900bc3f4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:09:20.489001 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7e25f8-8e38-456e-bcf1-18a6900bc3f4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:20.490582 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7e25f8-8e38-456e-bcf1-18a6900bc3f4 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:20.503554 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1273/2562] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:09:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 01:09:20.509433 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b57d42-7caa-4d60-ae22-0e136b1dd79a None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:20.523331 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b57d42-7caa-4d60-ae22-0e136b1dd79a None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:20.531927 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca177aaa-f006-42a2-998b-88fee4906790 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:20.532833 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca177aaa-f006-42a2-998b-88fee4906790 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:20.533229 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca177aaa-f006-42a2-998b-88fee4906790 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:20.533565 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca177aaa-f006-42a2-998b-88fee4906790 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:20.534565 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca177aaa-f006-42a2-998b-88fee4906790 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:20.536819 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca177aaa-f006-42a2-998b-88fee4906790 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:20.568653 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1290/2563] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:09:20 2022] POST /identity/v3/projects => generated 373 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:09:20.573029 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7e25f8-8e38-456e-bcf1-18a6900bc3f4 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:20.584368 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7e25f8-8e38-456e-bcf1-18a6900bc3f4 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:20.641901 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b100cb4-ef4f-42c6-bce3-83ccb51eac81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:20.642434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b100cb4-ef4f-42c6-bce3-83ccb51eac81 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:20.642702 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b100cb4-ef4f-42c6-bce3-83ccb51eac81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:20.642963 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b100cb4-ef4f-42c6-bce3-83ccb51eac81 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:20.643626 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b100cb4-ef4f-42c6-bce3-83ccb51eac81 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:20.645648 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b100cb4-ef4f-42c6-bce3-83ccb51eac81 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:20.673904 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1274/2564] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:09:20 2022] POST /identity/v3/users => generated 287 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 30 01:09:20.680096 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca177aaa-f006-42a2-998b-88fee4906790 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:20.695665 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca177aaa-f006-42a2-998b-88fee4906790 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:20.703204 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc526bf1-4d2e-483f-884a-cdaaf5012549 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:20.704053 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc526bf1-4d2e-483f-884a-cdaaf5012549 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:20.704351 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc526bf1-4d2e-483f-884a-cdaaf5012549 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:20.704613 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc526bf1-4d2e-483f-884a-cdaaf5012549 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:20.705439 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc526bf1-4d2e-483f-884a-cdaaf5012549 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:20.707585 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc526bf1-4d2e-483f-884a-cdaaf5012549 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:20.718267 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1291/2565] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:20 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:09:20.722415 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b100cb4-ef4f-42c6-bce3-83ccb51eac81 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:20.732955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b100cb4-ef4f-42c6-bce3-83ccb51eac81 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:20.738242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980c2ce1-751c-46c3-ac50-a663e5a7256d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:20.738830 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980c2ce1-751c-46c3-ac50-a663e5a7256d admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:20.739106 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980c2ce1-751c-46c3-ac50-a663e5a7256d admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:20.739365 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980c2ce1-751c-46c3-ac50-a663e5a7256d admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:20.740233 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980c2ce1-751c-46c3-ac50-a663e5a7256d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:20.741813 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980c2ce1-751c-46c3-ac50-a663e5a7256d admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:20.752594 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1275/2566] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:09:20.756603 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc526bf1-4d2e-483f-884a-cdaaf5012549 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:20.766565 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc526bf1-4d2e-483f-884a-cdaaf5012549 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:20.771781 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4a9ac9-c118-40bc-883d-bff6babd06f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:20.772431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4a9ac9-c118-40bc-883d-bff6babd06f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:20.772701 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4a9ac9-c118-40bc-883d-bff6babd06f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:20.772967 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4a9ac9-c118-40bc-883d-bff6babd06f3 admin admin] PATH_INFO: `/v3/projects/572e07ef2169453dae2f14ef8c2fb03d/users/d4977a0a41c04e958bd88139f8788d66/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:20.787429 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4a9ac9-c118-40bc-883d-bff6babd06f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=572e07ef2169453dae2f14ef8c2fb03d, user_id=d4977a0a41c04e958bd88139f8788d66, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:20.790076 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4a9ac9-c118-40bc-883d-bff6babd06f3 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:20.805690 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1292/2567] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:09:20 2022] PUT /identity/v3/projects/572e07ef2169453dae2f14ef8c2fb03d/users/d4977a0a41c04e958bd88139f8788d66/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:20.817814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a998e44-4cdc-46c0-8f52-1ba6d7fdd6f9 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:20.818096 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a998e44-4cdc-46c0-8f52-1ba6d7fdd6f9 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:20.818480 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a998e44-4cdc-46c0-8f52-1ba6d7fdd6f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:20.849955 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-4a998e44-4cdc-46c0-8f52-1ba6d7fdd6f9 None None] MFA Rules not processed for user `d4977a0a41c04e958bd88139f8788d66`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:20.851330 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-4a998e44-4cdc-46c0-8f52-1ba6d7fdd6f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:20.924615 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1276/2568] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:20 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:20.939996 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4a9ac9-c118-40bc-883d-bff6babd06f3 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:20.941945 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-9c4a9ac9-c118-40bc-883d-bff6babd06f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:21.016826 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4a9ac9-c118-40bc-883d-bff6babd06f3 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:21.021768 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ad67ee-8cb7-4ca5-aff6-a8b4dfc7cb07 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:21.022379 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ad67ee-8cb7-4ca5-aff6-a8b4dfc7cb07 service cinder] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:21.022651 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ad67ee-8cb7-4ca5-aff6-a8b4dfc7cb07 service cinder] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:21.022915 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ad67ee-8cb7-4ca5-aff6-a8b4dfc7cb07 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:21.032872 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ad67ee-8cb7-4ca5-aff6-a8b4dfc7cb07 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:21.034309 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ad67ee-8cb7-4ca5-aff6-a8b4dfc7cb07 service cinder] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:21.044119 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1293/2569] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:09:20 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:09:30.920166 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=96,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:31.040060 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=68,keystone:DELETE=6,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:31.552221 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a998e44-4cdc-46c0-8f52-1ba6d7fdd6f9 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:31.621663 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a998e44-4cdc-46c0-8f52-1ba6d7fdd6f9 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:31.626920 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f584e41-2dbc-4baf-9fea-79cdbd726648 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:31.627581 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f584e41-2dbc-4baf-9fea-79cdbd726648 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:31.627888 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f584e41-2dbc-4baf-9fea-79cdbd726648 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:31.628159 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f584e41-2dbc-4baf-9fea-79cdbd726648 admin admin] PATH_INFO: `/v3/users/d4977a0a41c04e958bd88139f8788d66` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:31.629323 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f584e41-2dbc-4baf-9fea-79cdbd726648 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d4977a0a41c04e958bd88139f8788d66)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:31.631280 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f584e41-2dbc-4baf-9fea-79cdbd726648 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:31.687177 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-9f584e41-2dbc-4baf-9fea-79cdbd726648 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd4977a0a41c04e958bd88139f8788d66'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:31.695186 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-9f584e41-2dbc-4baf-9fea-79cdbd726648 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd4977a0a41c04e958bd88139f8788d66'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:31.695903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-9f584e41-2dbc-4baf-9fea-79cdbd726648 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd4977a0a41c04e958bd88139f8788d66'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:31.703381 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-9f584e41-2dbc-4baf-9fea-79cdbd726648 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd4977a0a41c04e958bd88139f8788d66'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:31.715808 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-9f584e41-2dbc-4baf-9fea-79cdbd726648 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd4977a0a41c04e958bd88139f8788d66'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:31.721643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1277/2570] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:09:31 2022] DELETE /identity/v3/users/d4977a0a41c04e958bd88139f8788d66 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:31.726395 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da14197-ab3d-4f18-9853-fdc91322b7b4 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:31.726677 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da14197-ab3d-4f18-9853-fdc91322b7b4 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:31.726949 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da14197-ab3d-4f18-9853-fdc91322b7b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:31.747783 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-6da14197-ab3d-4f18-9853-fdc91322b7b4 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:31.749016 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-6da14197-ab3d-4f18-9853-fdc91322b7b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:31.811171 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1294/2571] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:31.821411 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f584e41-2dbc-4baf-9fea-79cdbd726648 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:31.823090 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-9f584e41-2dbc-4baf-9fea-79cdbd726648 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:31.885003 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f584e41-2dbc-4baf-9fea-79cdbd726648 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:31.889496 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd30efc-c005-4705-b6bc-5af741391858 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:31.890405 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd30efc-c005-4705-b6bc-5af741391858 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:31.890763 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd30efc-c005-4705-b6bc-5af741391858 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:31.891213 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd30efc-c005-4705-b6bc-5af741391858 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:31.904908 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd30efc-c005-4705-b6bc-5af741391858 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:31.906371 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd30efc-c005-4705-b6bc-5af741391858 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:31.916482 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1278/2572] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:09:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:09:32.185602 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da14197-ab3d-4f18-9853-fdc91322b7b4 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:32.187149 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-6da14197-ab3d-4f18-9853-fdc91322b7b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:32.199755 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da14197-ab3d-4f18-9853-fdc91322b7b4 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:32.205385 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e67819-504f-42be-84d3-db57164a5d67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:32.206900 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e67819-504f-42be-84d3-db57164a5d67 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:32.207199 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e67819-504f-42be-84d3-db57164a5d67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:32.207423 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e67819-504f-42be-84d3-db57164a5d67 admin admin] PATH_INFO: `/v3/projects/572e07ef2169453dae2f14ef8c2fb03d` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:32.209055 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e67819-504f-42be-84d3-db57164a5d67 admin admin] RBAC: Authorizing `identity:delete_project(project_id=572e07ef2169453dae2f14ef8c2fb03d)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:32.211138 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e67819-504f-42be-84d3-db57164a5d67 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:32.260674 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-50e67819-504f-42be-84d3-db57164a5d67 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '572e07ef2169453dae2f14ef8c2fb03d', 'request_id': 'req-50e67819-504f-42be-84d3-db57164a5d67'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:32.268657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-50e67819-504f-42be-84d3-db57164a5d67 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '572e07ef2169453dae2f14ef8c2fb03d', 'request_id': 'req-50e67819-504f-42be-84d3-db57164a5d67'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:32.274501 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-50e67819-504f-42be-84d3-db57164a5d67 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '572e07ef2169453dae2f14ef8c2fb03d', 'request_id': 'req-50e67819-504f-42be-84d3-db57164a5d67'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:32.286043 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-50e67819-504f-42be-84d3-db57164a5d67 admin admin] The token cache is being invalidate because project 572e07ef2169453dae2f14ef8c2fb03d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:09:32.286324 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-50e67819-504f-42be-84d3-db57164a5d67 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:32.287108 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-50e67819-504f-42be-84d3-db57164a5d67 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:09:32.288672 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1295/2573] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:09:32 2022] DELETE /identity/v3/projects/572e07ef2169453dae2f14ef8c2fb03d => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:32.294203 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9423014e-2338-43a8-a87c-dfd2fc2113c0 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:32.294691 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9423014e-2338-43a8-a87c-dfd2fc2113c0 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:32.294807 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9423014e-2338-43a8-a87c-dfd2fc2113c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:32.315375 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-9423014e-2338-43a8-a87c-dfd2fc2113c0 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:32.316616 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-9423014e-2338-43a8-a87c-dfd2fc2113c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:32.335252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:32.380631 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1279/2574] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:32.389193 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c346d557-45a5-4552-a94f-506512075d10 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:32.389499 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c346d557-45a5-4552-a94f-506512075d10 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:32.389767 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c346d557-45a5-4552-a94f-506512075d10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:32.409717 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-c346d557-45a5-4552-a94f-506512075d10 None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:32.410902 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-c346d557-45a5-4552-a94f-506512075d10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:32.419743 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1296/2575] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:32.423985 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9423014e-2338-43a8-a87c-dfd2fc2113c0 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:32.434197 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9423014e-2338-43a8-a87c-dfd2fc2113c0 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:32.439898 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73c3a4e-1f82-4536-9d1e-eb4cd153b4b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:32.440753 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73c3a4e-1f82-4536-9d1e-eb4cd153b4b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:32.441074 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73c3a4e-1f82-4536-9d1e-eb4cd153b4b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:32.441374 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73c3a4e-1f82-4536-9d1e-eb4cd153b4b8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:32.442370 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73c3a4e-1f82-4536-9d1e-eb4cd153b4b8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81494) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 30 01:09:32.442752 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73c3a4e-1f82-4536-9d1e-eb4cd153b4b8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:32.445104 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73c3a4e-1f82-4536-9d1e-eb4cd153b4b8 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:32.465318 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1280/2576] 10.222.0.16 () {58 vars in 1153 bytes} [Sun Oct 30 01:09:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 30 01:09:32.471432 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c346d557-45a5-4552-a94f-506512075d10 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:32.481681 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c346d557-45a5-4552-a94f-506512075d10 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:32.487205 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754c88cf-0dfc-465c-8ec9-c84a93fbd360 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:32.487789 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754c88cf-0dfc-465c-8ec9-c84a93fbd360 admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:32.488089 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754c88cf-0dfc-465c-8ec9-c84a93fbd360 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:32.488346 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754c88cf-0dfc-465c-8ec9-c84a93fbd360 admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:32.488971 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754c88cf-0dfc-465c-8ec9-c84a93fbd360 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:32.490860 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754c88cf-0dfc-465c-8ec9-c84a93fbd360 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:32.522624 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1297/2577] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:09:32 2022] POST /identity/v3/projects => generated 373 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:09:32.526984 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73c3a4e-1f82-4536-9d1e-eb4cd153b4b8 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:32.536891 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73c3a4e-1f82-4536-9d1e-eb4cd153b4b8 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:32.595749 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12215e14-3784-4a0b-9347-58cd0662d5a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:32.596646 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12215e14-3784-4a0b-9347-58cd0662d5a5 admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:32.597009 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12215e14-3784-4a0b-9347-58cd0662d5a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:32.597362 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12215e14-3784-4a0b-9347-58cd0662d5a5 admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:32.598301 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12215e14-3784-4a0b-9347-58cd0662d5a5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:32.601338 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12215e14-3784-4a0b-9347-58cd0662d5a5 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:32.628489 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1281/2578] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:09:32 2022] POST /identity/v3/users => generated 287 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:09:32.632723 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754c88cf-0dfc-465c-8ec9-c84a93fbd360 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:32.643144 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754c88cf-0dfc-465c-8ec9-c84a93fbd360 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:32.647851 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffb7730-79b6-4cad-9e08-e805dba3c6d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:32.648444 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffb7730-79b6-4cad-9e08-e805dba3c6d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:32.648718 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffb7730-79b6-4cad-9e08-e805dba3c6d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:32.649037 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffb7730-79b6-4cad-9e08-e805dba3c6d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:32.649637 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ffb7730-79b6-4cad-9e08-e805dba3c6d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:32.651220 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ffb7730-79b6-4cad-9e08-e805dba3c6d8 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:32.661176 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1298/2579] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:32 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:09:32.665320 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12215e14-3784-4a0b-9347-58cd0662d5a5 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:32.675704 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12215e14-3784-4a0b-9347-58cd0662d5a5 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:32.680897 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db541c04-28d5-419c-80c0-bef2e12de418 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:32.681486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db541c04-28d5-419c-80c0-bef2e12de418 admin admin] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:32.681761 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db541c04-28d5-419c-80c0-bef2e12de418 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:32.682028 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db541c04-28d5-419c-80c0-bef2e12de418 admin admin] PATH_INFO: `/v3/roles` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:32.682665 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db541c04-28d5-419c-80c0-bef2e12de418 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:32.684384 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db541c04-28d5-419c-80c0-bef2e12de418 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:32.695141 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1282/2580] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:32 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:09:32.699408 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffb7730-79b6-4cad-9e08-e805dba3c6d8 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:32.709991 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffb7730-79b6-4cad-9e08-e805dba3c6d8 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:32.715142 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c095743-bc52-45dc-8a62-bde57a3ef731 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:32.715894 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c095743-bc52-45dc-8a62-bde57a3ef731 admin admin] REQUEST_METHOD: `PUT` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:32.716218 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c095743-bc52-45dc-8a62-bde57a3ef731 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:32.716485 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c095743-bc52-45dc-8a62-bde57a3ef731 admin admin] PATH_INFO: `/v3/projects/4f281bf8af484f7a88df7dc8ac06d887/users/baeaad0029f1455497fc3aea467dfc47/roles/5c19ef44708c4d1f95f5f657c5c79361` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:32.734172 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c095743-bc52-45dc-8a62-bde57a3ef731 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4f281bf8af484f7a88df7dc8ac06d887, user_id=baeaad0029f1455497fc3aea467dfc47, role_id=5c19ef44708c4d1f95f5f657c5c79361)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:32.736831 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c095743-bc52-45dc-8a62-bde57a3ef731 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:32.752548 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1299/2581] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:09:32 2022] PUT /identity/v3/projects/4f281bf8af484f7a88df7dc8ac06d887/users/baeaad0029f1455497fc3aea467dfc47/roles/5c19ef44708c4d1f95f5f657c5c79361 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:32.757836 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1c4e56-7fa4-4a44-9730-9fb3827ddfea None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:32.758139 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1c4e56-7fa4-4a44-9730-9fb3827ddfea None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:32.758477 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1c4e56-7fa4-4a44-9730-9fb3827ddfea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:32.778587 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-8b1c4e56-7fa4-4a44-9730-9fb3827ddfea None None] MFA Rules not processed for user `ca6c0649e64b4d38bdd14a04f31beb73`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:32.779809 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-8b1c4e56-7fa4-4a44-9730-9fb3827ddfea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:32.845824 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1283/2582] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:32.855970 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c095743-bc52-45dc-8a62-bde57a3ef731 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:32.857701 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-1c095743-bc52-45dc-8a62-bde57a3ef731 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:32.880356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:32.916986 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c095743-bc52-45dc-8a62-bde57a3ef731 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:32.920722 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8a5683-10c8-496f-9c72-9e275ab2a8c6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:32.921317 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8a5683-10c8-496f-9c72-9e275ab2a8c6 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:32.921585 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8a5683-10c8-496f-9c72-9e275ab2a8c6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:32.921843 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8a5683-10c8-496f-9c72-9e275ab2a8c6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:32.931360 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8a5683-10c8-496f-9c72-9e275ab2a8c6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:32.932809 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8a5683-10c8-496f-9c72-9e275ab2a8c6 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:32.942606 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1300/2583] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:09:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:09:36.895512 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1ca47c-9e25-4bff-945d-fa1623c68446 None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:36.896082 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1ca47c-9e25-4bff-945d-fa1623c68446 None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:36.896082 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1ca47c-9e25-4bff-945d-fa1623c68446 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:36.925582 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-ed1ca47c-9e25-4bff-945d-fa1623c68446 None None] MFA Rules not processed for user `baeaad0029f1455497fc3aea467dfc47`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:36.926923 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-ed1ca47c-9e25-4bff-945d-fa1623c68446 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:36.948279 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:37.006240 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1284/2584] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:36 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:37.011026 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8a5683-10c8-496f-9c72-9e275ab2a8c6 service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:37.021366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8a5683-10c8-496f-9c72-9e275ab2a8c6 service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:37.026297 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22acea7-b2df-4e94-8a01-44916aed140c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:37.026910 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22acea7-b2df-4e94-8a01-44916aed140c admin admin] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:37.027183 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22acea7-b2df-4e94-8a01-44916aed140c admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:37.027452 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22acea7-b2df-4e94-8a01-44916aed140c admin admin] PATH_INFO: `/v3/projects` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:37.028126 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22acea7-b2df-4e94-8a01-44916aed140c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:37.030300 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22acea7-b2df-4e94-8a01-44916aed140c admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:37.062125 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1301/2585] 10.222.0.16 () {60 vars in 1153 bytes} [Sun Oct 30 01:09:37 2022] POST /identity/v3/projects => generated 373 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:09:37.063670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1ca47c-9e25-4bff-945d-fa1623c68446 None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:37.075182 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1ca47c-9e25-4bff-945d-fa1623c68446 None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:37.138919 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510ad414-3dbd-41e7-abc6-61a350ff500b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:37.139535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510ad414-3dbd-41e7-abc6-61a350ff500b admin admin] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:37.139814 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510ad414-3dbd-41e7-abc6-61a350ff500b admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:37.140068 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510ad414-3dbd-41e7-abc6-61a350ff500b admin admin] PATH_INFO: `/v3/users` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:37.140710 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510ad414-3dbd-41e7-abc6-61a350ff500b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:37.142635 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510ad414-3dbd-41e7-abc6-61a350ff500b admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:37.166033 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1285/2586] 10.222.0.16 () {60 vars in 1144 bytes} [Sun Oct 30 01:09:37 2022] POST /identity/v3/users => generated 293 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 30 01:09:37.170131 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22acea7-b2df-4e94-8a01-44916aed140c admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:37.180558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22acea7-b2df-4e94-8a01-44916aed140c admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:37.185564 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73c92e1-8544-4062-802c-ea445be9fc17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:37.186155 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73c92e1-8544-4062-802c-ea445be9fc17 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:37.186424 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73c92e1-8544-4062-802c-ea445be9fc17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:37.186692 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73c92e1-8544-4062-802c-ea445be9fc17 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:37.187429 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73c92e1-8544-4062-802c-ea445be9fc17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:37.189034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73c92e1-8544-4062-802c-ea445be9fc17 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:37.199985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1302/2587] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:37 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:09:37.204202 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510ad414-3dbd-41e7-abc6-61a350ff500b admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:37.215218 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510ad414-3dbd-41e7-abc6-61a350ff500b admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:37.219959 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef9085d-e74f-4c05-a5b9-46d728d56c30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:37.220600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef9085d-e74f-4c05-a5b9-46d728d56c30 admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:37.220873 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef9085d-e74f-4c05-a5b9-46d728d56c30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:37.221142 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef9085d-e74f-4c05-a5b9-46d728d56c30 admin admin] PATH_INFO: `/v3/projects/97a2499d78e44464bc2e9c8c2f9f145c/users/842d57c1685b4188882f3f796d7f3ae8/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:37.236918 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef9085d-e74f-4c05-a5b9-46d728d56c30 admin admin] RBAC: Authorizing `identity:create_grant(project_id=97a2499d78e44464bc2e9c8c2f9f145c, user_id=842d57c1685b4188882f3f796d7f3ae8, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:37.239556 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef9085d-e74f-4c05-a5b9-46d728d56c30 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:37.254457 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1286/2588] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:09:37 2022] PUT /identity/v3/projects/97a2499d78e44464bc2e9c8c2f9f145c/users/842d57c1685b4188882f3f796d7f3ae8/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:37.259999 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73c92e1-8544-4062-802c-ea445be9fc17 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:37.271494 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73c92e1-8544-4062-802c-ea445be9fc17 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:37.329846 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfecf72f-2519-4bdf-8c5b-a2c54b746eb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:37.330495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfecf72f-2519-4bdf-8c5b-a2c54b746eb8 admin admin] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:37.330751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfecf72f-2519-4bdf-8c5b-a2c54b746eb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:37.331016 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfecf72f-2519-4bdf-8c5b-a2c54b746eb8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:37.331652 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfecf72f-2519-4bdf-8c5b-a2c54b746eb8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:37.333654 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfecf72f-2519-4bdf-8c5b-a2c54b746eb8 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:37.348010 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1303/2589] 10.222.0.16 () {58 vars in 1122 bytes} [Sun Oct 30 01:09:37 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 30 01:09:37.349563 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef9085d-e74f-4c05-a5b9-46d728d56c30 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:37.360550 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef9085d-e74f-4c05-a5b9-46d728d56c30 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:37.365478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f8d247-f8c0-4f1d-a967-e8395c3dabaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:37.366116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f8d247-f8c0-4f1d-a967-e8395c3dabaa admin admin] REQUEST_METHOD: `PUT` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:37.366384 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f8d247-f8c0-4f1d-a967-e8395c3dabaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:37.366650 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f8d247-f8c0-4f1d-a967-e8395c3dabaa admin admin] PATH_INFO: `/v3/projects/97a2499d78e44464bc2e9c8c2f9f145c/users/842d57c1685b4188882f3f796d7f3ae8/roles/35f59686b5d84291a8d9baf19287eb5a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:37.368798 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f8d247-f8c0-4f1d-a967-e8395c3dabaa admin admin] RBAC: Authorizing `identity:create_grant(project_id=97a2499d78e44464bc2e9c8c2f9f145c, user_id=842d57c1685b4188882f3f796d7f3ae8, role_id=35f59686b5d84291a8d9baf19287eb5a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:37.371407 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f8d247-f8c0-4f1d-a967-e8395c3dabaa admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:37.382643 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1287/2590] 10.222.0.16 () {60 vars in 1483 bytes} [Sun Oct 30 01:09:37 2022] PUT /identity/v3/projects/97a2499d78e44464bc2e9c8c2f9f145c/users/842d57c1685b4188882f3f796d7f3ae8/roles/35f59686b5d84291a8d9baf19287eb5a => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:09:41.754273 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41415c15-81ec-4c28-a961-19bbd95a74fd None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:41.754751 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41415c15-81ec-4c28-a961-19bbd95a74fd None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:41.754934 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41415c15-81ec-4c28-a961-19bbd95a74fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:41.790649 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-41415c15-81ec-4c28-a961-19bbd95a74fd None None] MFA Rules not processed for user `842d57c1685b4188882f3f796d7f3ae8`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:41.792063 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-41415c15-81ec-4c28-a961-19bbd95a74fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:41.890004 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1304/2591] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:41 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:41.905556 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f8d247-f8c0-4f1d-a967-e8395c3dabaa admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:09:41.907416 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-36f8d247-f8c0-4f1d-a967-e8395c3dabaa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:41.985389 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f8d247-f8c0-4f1d-a967-e8395c3dabaa admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:09:41.991288 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3e97b4-3561-4e05-bd79-d4842e7fa547 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:09:41.992095 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3e97b4-3561-4e05-bd79-d4842e7fa547 service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:41.992422 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3e97b4-3561-4e05-bd79-d4842e7fa547 service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:41.992785 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3e97b4-3561-4e05-bd79-d4842e7fa547 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:41.999845 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:42.055134 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3e97b4-3561-4e05-bd79-d4842e7fa547 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:09:42.056852 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3e97b4-3561-4e05-bd79-d4842e7fa547 service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:09:42.066417 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1288/2592] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:09:41 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 01:09:44.080546 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c83ecd0-3446-486c-8724-991de0aefe9a None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:09:44.081186 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c83ecd0-3446-486c-8724-991de0aefe9a None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:09:44.081186 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c83ecd0-3446-486c-8724-991de0aefe9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:09:44.102757 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-3c83ecd0-3446-486c-8724-991de0aefe9a None None] MFA Rules not processed for user `baeaad0029f1455497fc3aea467dfc47`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:09:44.104104 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3c83ecd0-3446-486c-8724-991de0aefe9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:09:44.111805 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1305/2593] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:09:44 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:09:52.062260 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=24 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:09:54.096023 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=97,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:10:07.422983 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3e97b4-3561-4e05-bd79-d4842e7fa547 service nova] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:10:07.425362 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-2a3e97b4-3561-4e05-bd79-d4842e7fa547 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:10:07.505498 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3e97b4-3561-4e05-bd79-d4842e7fa547 service nova] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:10:07.511310 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28484058-6d3d-4ade-b1dd-1d457992f684 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '63cc035057064805a95e9f12714ebade', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:10:07.511954 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28484058-6d3d-4ade-b1dd-1d457992f684 service placement] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:10:07.512232 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28484058-6d3d-4ade-b1dd-1d457992f684 service placement] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:10:07.512498 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28484058-6d3d-4ade-b1dd-1d457992f684 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:10:07.514314 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-28484058-6d3d-4ade-b1dd-1d457992f684 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:10:07.525937 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28484058-6d3d-4ade-b1dd-1d457992f684 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:10:07.527609 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28484058-6d3d-4ade-b1dd-1d457992f684 service placement] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:10:07.537892 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1289/2594] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:10:07 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:10:17.533997 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=32 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:10:24.832055 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c83ecd0-3446-486c-8724-991de0aefe9a None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:10:24.833810 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3c83ecd0-3446-486c-8724-991de0aefe9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:10:24.900650 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c83ecd0-3446-486c-8724-991de0aefe9a None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:10:24.904584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ae0fdd-7963-4ce6-afd1-a447934bc6ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:10:24.905191 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ae0fdd-7963-4ce6-afd1-a447934bc6ac service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:10:24.905463 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ae0fdd-7963-4ce6-afd1-a447934bc6ac service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:10:24.905728 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ae0fdd-7963-4ce6-afd1-a447934bc6ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:10:24.915050 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ae0fdd-7963-4ce6-afd1-a447934bc6ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:10:24.916534 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ae0fdd-7963-4ce6-afd1-a447934bc6ac service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:10:24.926322 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1306/2595] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:10:24 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:10:26.567470 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28484058-6d3d-4ade-b1dd-1d457992f684 service placement] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:10:26.568975 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-28484058-6d3d-4ade-b1dd-1d457992f684 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:10:26.584837 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28484058-6d3d-4ade-b1dd-1d457992f684 service placement] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:10:26.590826 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92a858c-2aac-4577-ba79-89dd4e2df29d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:10:26.591434 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92a858c-2aac-4577-ba79-89dd4e2df29d service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:10:26.591703 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92a858c-2aac-4577-ba79-89dd4e2df29d service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:10:26.592004 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92a858c-2aac-4577-ba79-89dd4e2df29d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:10:26.593705 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b92a858c-2aac-4577-ba79-89dd4e2df29d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:10:26.606579 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92a858c-2aac-4577-ba79-89dd4e2df29d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:10:26.608154 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92a858c-2aac-4577-ba79-89dd4e2df29d service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:10:26.619423 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1290/2596] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:10:26 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:10:34.921926 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=26 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:10:36.614645 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=6 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:10:44.233957 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a902f64-61cb-406c-813d-321a5365fdb4 None None] REQUEST_METHOD: `POST` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:10:44.234470 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a902f64-61cb-406c-813d-321a5365fdb4 None None] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:10:44.234470 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a902f64-61cb-406c-813d-321a5365fdb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:10:44.254545 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.auth.core [None req-3a902f64-61cb-406c-813d-321a5365fdb4 None None] MFA Rules not processed for user `96e0cd00e4ef4a2da960aa5d4f66972a`. Rule list: `[]` (Enabled: `True`). {{(pid=81495) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:10:44.255747 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-3a902f64-61cb-406c-813d-321a5365fdb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:10:44.314603 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1307/2597] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:10:44 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:10:47.274027 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92a858c-2aac-4577-ba79-89dd4e2df29d service nova] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:10:47.283755 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92a858c-2aac-4577-ba79-89dd4e2df29d service nova] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:10:47.287834 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56c69ac-079f-41ad-aa08-1f272442fc1d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:10:47.288511 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56c69ac-079f-41ad-aa08-1f272442fc1d service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:10:47.288845 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56c69ac-079f-41ad-aa08-1f272442fc1d service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:10:47.289171 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56c69ac-079f-41ad-aa08-1f272442fc1d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:10:47.290965 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-f56c69ac-079f-41ad-aa08-1f272442fc1d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:10:47.304524 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56c69ac-079f-41ad-aa08-1f272442fc1d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:10:47.306014 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56c69ac-079f-41ad-aa08-1f272442fc1d service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:10:47.317472 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1291/2598] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:10:47 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:10:53.187641 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a902f64-61cb-406c-813d-321a5365fdb4 None None] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:10:53.198876 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a902f64-61cb-406c-813d-321a5365fdb4 None None] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:10:53.204421 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009ab90c-da59-4de1-9c6c-54ea4cb2f12b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:10:53.205056 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009ab90c-da59-4de1-9c6c-54ea4cb2f12b service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:10:53.205323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009ab90c-da59-4de1-9c6c-54ea4cb2f12b service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:10:53.205584 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009ab90c-da59-4de1-9c6c-54ea4cb2f12b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:10:53.208074 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-009ab90c-da59-4de1-9c6c-54ea4cb2f12b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:10:53.223536 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009ab90c-da59-4de1-9c6c-54ea4cb2f12b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:10:53.225034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009ab90c-da59-4de1-9c6c-54ea4cb2f12b service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:10:53.236298 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1308/2599] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:10:53 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:10:57.312369 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=6 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:11:03.231256 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=31 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:11:12.421033 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56c69ac-079f-41ad-aa08-1f272442fc1d service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:11:12.431383 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56c69ac-079f-41ad-aa08-1f272442fc1d service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:11:12.436574 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24206935-d17d-4b26-be49-ed75d51ffc04 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:11:12.437173 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24206935-d17d-4b26-be49-ed75d51ffc04 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:11:12.437444 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24206935-d17d-4b26-be49-ed75d51ffc04 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:11:12.437709 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24206935-d17d-4b26-be49-ed75d51ffc04 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:11:12.439641 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-24206935-d17d-4b26-be49-ed75d51ffc04 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:11:12.450098 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24206935-d17d-4b26-be49-ed75d51ffc04 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:11:12.451560 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24206935-d17d-4b26-be49-ed75d51ffc04 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:11:12.460306 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1292/2600] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:11:12 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:11:22.456351 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=6 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:12:56.996721 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009ab90c-da59-4de1-9c6c-54ea4cb2f12b service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:12:57.007491 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009ab90c-da59-4de1-9c6c-54ea4cb2f12b service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:12:57.011659 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6731cd36-7805-4c3f-a7ed-aff64f0fb676 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:12:57.012295 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6731cd36-7805-4c3f-a7ed-aff64f0fb676 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:12:57.012658 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6731cd36-7805-4c3f-a7ed-aff64f0fb676 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:12:57.012913 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6731cd36-7805-4c3f-a7ed-aff64f0fb676 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:12:57.014870 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-6731cd36-7805-4c3f-a7ed-aff64f0fb676 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:12:57.026170 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6731cd36-7805-4c3f-a7ed-aff64f0fb676 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:12:57.027633 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6731cd36-7805-4c3f-a7ed-aff64f0fb676 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:12:57.036006 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1309/2601] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:12:56 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:12:57.119441 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24206935-d17d-4b26-be49-ed75d51ffc04 service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:12:57.128800 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24206935-d17d-4b26-be49-ed75d51ffc04 service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:12:57.132545 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd094a5f-ccab-43ca-8014-69da210d4554 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:12:57.133266 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd094a5f-ccab-43ca-8014-69da210d4554 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:12:57.133600 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd094a5f-ccab-43ca-8014-69da210d4554 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:12:57.134451 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd094a5f-ccab-43ca-8014-69da210d4554 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:12:57.136314 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-bd094a5f-ccab-43ca-8014-69da210d4554 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:12:57.150225 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd094a5f-ccab-43ca-8014-69da210d4554 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:12:57.151657 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd094a5f-ccab-43ca-8014-69da210d4554 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:12:57.162684 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1293/2602] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:12:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:12:57.230293 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6731cd36-7805-4c3f-a7ed-aff64f0fb676 service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:12:57.245185 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6731cd36-7805-4c3f-a7ed-aff64f0fb676 service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:12:57.250843 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818b4fcc-ac89-469e-a165-d9a7f49526e3 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '63cc035057064805a95e9f12714ebade', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:12:57.251478 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818b4fcc-ac89-469e-a165-d9a7f49526e3 service placement] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:12:57.251740 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818b4fcc-ac89-469e-a165-d9a7f49526e3 service placement] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:12:57.252054 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818b4fcc-ac89-469e-a165-d9a7f49526e3 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:12:57.253858 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-818b4fcc-ac89-469e-a165-d9a7f49526e3 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:12:57.265562 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818b4fcc-ac89-469e-a165-d9a7f49526e3 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:12:57.267244 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818b4fcc-ac89-469e-a165-d9a7f49526e3 service placement] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:12:57.277326 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1310/2603] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:12:57 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:13:07.158253 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=6 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:13:07.273100 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=12 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:13:07.590200 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd094a5f-ccab-43ca-8014-69da210d4554 service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:13:07.591876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-bd094a5f-ccab-43ca-8014-69da210d4554 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:13:07.652214 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd094a5f-ccab-43ca-8014-69da210d4554 service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:13:07.656269 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f5f88e-eb02-4f4c-8582-c197f5a6ba28 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd57b1c923dff462ab9aaf238f95c6b7f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:13:07.656876 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f5f88e-eb02-4f4c-8582-c197f5a6ba28 service cinder] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:13:07.657893 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f5f88e-eb02-4f4c-8582-c197f5a6ba28 service cinder] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:13:07.658198 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f5f88e-eb02-4f4c-8582-c197f5a6ba28 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:13:07.660461 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-37f5f88e-eb02-4f4c-8582-c197f5a6ba28 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:13:07.672263 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f5f88e-eb02-4f4c-8582-c197f5a6ba28 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:13:07.673755 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f5f88e-eb02-4f4c-8582-c197f5a6ba28 service cinder] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:13:07.683794 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1294/2604] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:13:07 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:13:13.957932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818b4fcc-ac89-469e-a165-d9a7f49526e3 service placement] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:13:13.972041 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818b4fcc-ac89-469e-a165-d9a7f49526e3 service placement] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:13:13.977923 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42fd422-431d-4dd5-ae57-921ba29795a7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:13:13.978538 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42fd422-431d-4dd5-ae57-921ba29795a7 service nova] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:13:13.978815 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42fd422-431d-4dd5-ae57-921ba29795a7 service nova] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:13:13.979085 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42fd422-431d-4dd5-ae57-921ba29795a7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:13:13.980894 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-d42fd422-431d-4dd5-ae57-921ba29795a7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:13:13.993411 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42fd422-431d-4dd5-ae57-921ba29795a7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:13:13.995016 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42fd422-431d-4dd5-ae57-921ba29795a7 service nova] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:13:14.006740 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1311/2605] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:13:13 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:13:17.678697 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=26 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:13:19.919011 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70183697-5cd9-453d-91bd-98b17bf5e38d None None] REQUEST_METHOD: `POST` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:13:19.919658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70183697-5cd9-453d-91bd-98b17bf5e38d None None] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:13:19.919658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70183697-5cd9-453d-91bd-98b17bf5e38d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:13:19.948658 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.auth.core [None req-70183697-5cd9-453d-91bd-98b17bf5e38d None None] MFA Rules not processed for user `96e0cd00e4ef4a2da960aa5d4f66972a`. Rule list: `[]` (Enabled: `True`). {{(pid=81494) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 30 01:13:19.949860 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-70183697-5cd9-453d-91bd-98b17bf5e38d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:13:19.958950 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1295/2606] 10.222.0.16 () {58 vars in 958 bytes} [Sun Oct 30 01:13:19 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 30 01:13:20.303595 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42fd422-431d-4dd5-ae57-921ba29795a7 service nova] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:13:20.313807 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42fd422-431d-4dd5-ae57-921ba29795a7 service nova] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:13:20.317632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e1f41b-69c6-4a7a-8f8a-547b71c68341 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:13:20.318232 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e1f41b-69c6-4a7a-8f8a-547b71c68341 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:13:20.318506 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e1f41b-69c6-4a7a-8f8a-547b71c68341 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:13:20.318773 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e1f41b-69c6-4a7a-8f8a-547b71c68341 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:13:20.320968 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-b8e1f41b-69c6-4a7a-8f8a-547b71c68341 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:13:20.331675 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e1f41b-69c6-4a7a-8f8a-547b71c68341 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:13:20.333130 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e1f41b-69c6-4a7a-8f8a-547b71c68341 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:13:20.341872 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1312/2607] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:13:20 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:13:24.914706 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70183697-5cd9-453d-91bd-98b17bf5e38d None None] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:13:24.930556 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70183697-5cd9-453d-91bd-98b17bf5e38d None None] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:13:24.936621 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666fb60c-f0c2-4e46-9e84-5a24ed9c0943 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:13:24.937238 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666fb60c-f0c2-4e46-9e84-5a24ed9c0943 service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:13:24.937513 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666fb60c-f0c2-4e46-9e84-5a24ed9c0943 service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:13:24.937781 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666fb60c-f0c2-4e46-9e84-5a24ed9c0943 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:13:24.939845 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-666fb60c-f0c2-4e46-9e84-5a24ed9c0943 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:13:24.953784 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666fb60c-f0c2-4e46-9e84-5a24ed9c0943 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:13:24.955393 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666fb60c-f0c2-4e46-9e84-5a24ed9c0943 service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:13:24.967159 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1296/2608] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:13:24 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:13:30.338111 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=12 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:13:34.961982 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=9 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:13:38.844522 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e1f41b-69c6-4a7a-8f8a-547b71c68341 service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:13:38.854268 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e1f41b-69c6-4a7a-8f8a-547b71c68341 service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:13:38.858559 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0432ec6e-f4e3-41a0-8a83-dce6ff9423a6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:13:38.859162 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0432ec6e-f4e3-41a0-8a83-dce6ff9423a6 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:13:38.859431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0432ec6e-f4e3-41a0-8a83-dce6ff9423a6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:13:38.859690 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0432ec6e-f4e3-41a0-8a83-dce6ff9423a6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:13:38.861853 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-0432ec6e-f4e3-41a0-8a83-dce6ff9423a6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:13:38.931323 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0432ec6e-f4e3-41a0-8a83-dce6ff9423a6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:13:38.933844 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0432ec6e-f4e3-41a0-8a83-dce6ff9423a6 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:13:38.944225 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1313/2609] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:13:38 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:13:48.940000 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=30 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:15:02.471015 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666fb60c-f0c2-4e46-9e84-5a24ed9c0943 service nova] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:15:02.492764 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666fb60c-f0c2-4e46-9e84-5a24ed9c0943 service nova] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:15:02.496877 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79861b8-af4b-4b30-95de-7ace432d4320 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:15:02.497483 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79861b8-af4b-4b30-95de-7ace432d4320 service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:15:02.497762 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79861b8-af4b-4b30-95de-7ace432d4320 service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:15:02.498026 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79861b8-af4b-4b30-95de-7ace432d4320 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:15:02.521361 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79861b8-af4b-4b30-95de-7ace432d4320 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:15:02.522777 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79861b8-af4b-4b30-95de-7ace432d4320 service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:15:02.531798 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1297/2610] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:15:02 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:15:08.433558 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0432ec6e-f4e3-41a0-8a83-dce6ff9423a6 service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:15:08.451749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0432ec6e-f4e3-41a0-8a83-dce6ff9423a6 service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:15:08.455495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ff88b2-6679-48e0-88ab-d9512e8a2bd6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:15:08.456185 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ff88b2-6679-48e0-88ab-d9512e8a2bd6 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:15:08.456447 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ff88b2-6679-48e0-88ab-d9512e8a2bd6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:15:08.456709 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ff88b2-6679-48e0-88ab-d9512e8a2bd6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:15:08.522893 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ff88b2-6679-48e0-88ab-d9512e8a2bd6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:15:08.524382 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ff88b2-6679-48e0-88ab-d9512e8a2bd6 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:15:08.534294 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1314/2611] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:15:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:15:11.931666 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79861b8-af4b-4b30-95de-7ace432d4320 service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:15:11.933219 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-a79861b8-af4b-4b30-95de-7ace432d4320 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:15:11.997241 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79861b8-af4b-4b30-95de-7ace432d4320 service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:15:12.003189 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d92c3c9-baa5-4410-86bd-544a48c9f851 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '63cc035057064805a95e9f12714ebade', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:15:12.004041 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d92c3c9-baa5-4410-86bd-544a48c9f851 service placement] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:15:12.004316 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d92c3c9-baa5-4410-86bd-544a48c9f851 service placement] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:15:12.004583 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d92c3c9-baa5-4410-86bd-544a48c9f851 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:15:12.006540 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-1d92c3c9-baa5-4410-86bd-544a48c9f851 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:15:12.019248 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d92c3c9-baa5-4410-86bd-544a48c9f851 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:15:12.020854 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d92c3c9-baa5-4410-86bd-544a48c9f851 service placement] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:15:12.031258 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1298/2612] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:15:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:15:15.164068 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ff88b2-6679-48e0-88ab-d9512e8a2bd6 service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:15:15.174095 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ff88b2-6679-48e0-88ab-d9512e8a2bd6 service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:15:15.179647 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a94c87-ade0-4060-92e3-e2f2d126c2b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:15:15.180698 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a94c87-ade0-4060-92e3-e2f2d126c2b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:15:15.180698 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a94c87-ade0-4060-92e3-e2f2d126c2b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:15:15.180923 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a94c87-ade0-4060-92e3-e2f2d126c2b8 admin admin] PATH_INFO: `/v3/users/baeaad0029f1455497fc3aea467dfc47` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:15:15.182031 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a94c87-ade0-4060-92e3-e2f2d126c2b8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=baeaad0029f1455497fc3aea467dfc47)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:15:15.184048 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a94c87-ade0-4060-92e3-e2f2d126c2b8 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:15:15.234693 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a5a94c87-ade0-4060-92e3-e2f2d126c2b8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'baeaad0029f1455497fc3aea467dfc47'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:15:15.245808 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a5a94c87-ade0-4060-92e3-e2f2d126c2b8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'baeaad0029f1455497fc3aea467dfc47'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:15:15.246852 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a5a94c87-ade0-4060-92e3-e2f2d126c2b8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'baeaad0029f1455497fc3aea467dfc47'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:15:15.260800 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a5a94c87-ade0-4060-92e3-e2f2d126c2b8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'baeaad0029f1455497fc3aea467dfc47'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:15:15.267775 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a5a94c87-ade0-4060-92e3-e2f2d126c2b8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'baeaad0029f1455497fc3aea467dfc47'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:15:15.274015 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1315/2613] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:15:15 2022] DELETE /identity/v3/users/baeaad0029f1455497fc3aea467dfc47 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:15:15.461486 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d92c3c9-baa5-4410-86bd-544a48c9f851 service placement] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:15:15.472257 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d92c3c9-baa5-4410-86bd-544a48c9f851 service placement] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:15:15.531025 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af3e587-2833-49f0-a06b-76e10b0c300a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:15:15.531554 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af3e587-2833-49f0-a06b-76e10b0c300a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:15:15.531862 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af3e587-2833-49f0-a06b-76e10b0c300a admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:15:15.532131 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af3e587-2833-49f0-a06b-76e10b0c300a admin admin] PATH_INFO: `/v3/projects/4f281bf8af484f7a88df7dc8ac06d887` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:15:15.533344 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af3e587-2833-49f0-a06b-76e10b0c300a admin admin] RBAC: Authorizing `identity:delete_project(project_id=4f281bf8af484f7a88df7dc8ac06d887)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:15:15.535363 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af3e587-2833-49f0-a06b-76e10b0c300a admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:15:15.585380 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-8af3e587-2833-49f0-a06b-76e10b0c300a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4f281bf8af484f7a88df7dc8ac06d887', 'request_id': 'req-8af3e587-2833-49f0-a06b-76e10b0c300a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:15:15.591804 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-8af3e587-2833-49f0-a06b-76e10b0c300a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4f281bf8af484f7a88df7dc8ac06d887', 'request_id': 'req-8af3e587-2833-49f0-a06b-76e10b0c300a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:15:15.603693 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-8af3e587-2833-49f0-a06b-76e10b0c300a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4f281bf8af484f7a88df7dc8ac06d887', 'request_id': 'req-8af3e587-2833-49f0-a06b-76e10b0c300a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:15:15.615328 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-8af3e587-2833-49f0-a06b-76e10b0c300a admin admin] The token cache is being invalidate because project 4f281bf8af484f7a88df7dc8ac06d887 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:15:15.615607 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-8af3e587-2833-49f0-a06b-76e10b0c300a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:15:15.616295 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-8af3e587-2833-49f0-a06b-76e10b0c300a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:15:15.617821 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1299/2614] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:15:15 2022] DELETE /identity/v3/projects/4f281bf8af484f7a88df7dc8ac06d887 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:15:15.622311 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a94c87-ade0-4060-92e3-e2f2d126c2b8 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:15:15.623959 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-a5a94c87-ade0-4060-92e3-e2f2d126c2b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:15:15.638512 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a94c87-ade0-4060-92e3-e2f2d126c2b8 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:15:15.686423 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:15:15.700684 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e970851e-f71c-4c7a-a0c7-1308e8b17d5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:15:15.701355 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e970851e-f71c-4c7a-a0c7-1308e8b17d5e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:15:15.701632 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e970851e-f71c-4c7a-a0c7-1308e8b17d5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:15:15.701900 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e970851e-f71c-4c7a-a0c7-1308e8b17d5e admin admin] PATH_INFO: `/v3/users/842d57c1685b4188882f3f796d7f3ae8` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:15:15.703398 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e970851e-f71c-4c7a-a0c7-1308e8b17d5e admin admin] RBAC: Authorizing `identity:delete_user(user_id=842d57c1685b4188882f3f796d7f3ae8)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:15:15.705357 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e970851e-f71c-4c7a-a0c7-1308e8b17d5e admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:15:15.756749 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-e970851e-f71c-4c7a-a0c7-1308e8b17d5e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '842d57c1685b4188882f3f796d7f3ae8'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:15:15.765685 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-e970851e-f71c-4c7a-a0c7-1308e8b17d5e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '842d57c1685b4188882f3f796d7f3ae8'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:15:15.766284 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-e970851e-f71c-4c7a-a0c7-1308e8b17d5e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '842d57c1685b4188882f3f796d7f3ae8'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:15:15.778186 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-e970851e-f71c-4c7a-a0c7-1308e8b17d5e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '842d57c1685b4188882f3f796d7f3ae8'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:15:15.786708 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-e970851e-f71c-4c7a-a0c7-1308e8b17d5e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '842d57c1685b4188882f3f796d7f3ae8'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:15:15.792824 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1316/2615] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:15:15 2022] DELETE /identity/v3/users/842d57c1685b4188882f3f796d7f3ae8 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:15:15.968629 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af3e587-2833-49f0-a06b-76e10b0c300a admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:15:15.970898 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-8af3e587-2833-49f0-a06b-76e10b0c300a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:15:15.980077 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:15:15.987539 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af3e587-2833-49f0-a06b-76e10b0c300a admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:15:16.047523 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cf6dd0-6772-48ab-b1c1-8ce968922593 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:15:16.048100 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cf6dd0-6772-48ab-b1c1-8ce968922593 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:15:16.048368 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cf6dd0-6772-48ab-b1c1-8ce968922593 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:15:16.048627 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cf6dd0-6772-48ab-b1c1-8ce968922593 admin admin] PATH_INFO: `/v3/projects/97a2499d78e44464bc2e9c8c2f9f145c` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:15:16.049722 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3cf6dd0-6772-48ab-b1c1-8ce968922593 admin admin] RBAC: Authorizing `identity:delete_project(project_id=97a2499d78e44464bc2e9c8c2f9f145c)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:15:16.051807 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3cf6dd0-6772-48ab-b1c1-8ce968922593 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:15:16.108995 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-d3cf6dd0-6772-48ab-b1c1-8ce968922593 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '97a2499d78e44464bc2e9c8c2f9f145c', 'request_id': 'req-d3cf6dd0-6772-48ab-b1c1-8ce968922593'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:15:16.115242 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-d3cf6dd0-6772-48ab-b1c1-8ce968922593 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '97a2499d78e44464bc2e9c8c2f9f145c', 'request_id': 'req-d3cf6dd0-6772-48ab-b1c1-8ce968922593'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:15:16.124525 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-d3cf6dd0-6772-48ab-b1c1-8ce968922593 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '97a2499d78e44464bc2e9c8c2f9f145c', 'request_id': 'req-d3cf6dd0-6772-48ab-b1c1-8ce968922593'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:15:16.136499 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-d3cf6dd0-6772-48ab-b1c1-8ce968922593 admin admin] The token cache is being invalidate because project 97a2499d78e44464bc2e9c8c2f9f145c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81494) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:15:16.136782 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-d3cf6dd0-6772-48ab-b1c1-8ce968922593 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:15:16.137488 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-d3cf6dd0-6772-48ab-b1c1-8ce968922593 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:15:16.139122 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1300/2616] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:15:15 2022] DELETE /identity/v3/projects/97a2499d78e44464bc2e9c8c2f9f145c => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:15:16.272248 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e970851e-f71c-4c7a-a0c7-1308e8b17d5e admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:15:16.273886 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-e970851e-f71c-4c7a-a0c7-1308e8b17d5e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:15:16.352396 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e970851e-f71c-4c7a-a0c7-1308e8b17d5e admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:15:16.423101 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461c98a6-69f8-4e62-89f5-19caa5a8c0a1 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '63cc035057064805a95e9f12714ebade', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:15:16.423728 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461c98a6-69f8-4e62-89f5-19caa5a8c0a1 service placement] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:15:16.424097 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461c98a6-69f8-4e62-89f5-19caa5a8c0a1 service placement] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:15:16.424366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461c98a6-69f8-4e62-89f5-19caa5a8c0a1 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:15:16.426201 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-461c98a6-69f8-4e62-89f5-19caa5a8c0a1 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:15:16.437431 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461c98a6-69f8-4e62-89f5-19caa5a8c0a1 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:15:16.438992 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461c98a6-69f8-4e62-89f5-19caa5a8c0a1 service placement] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:15:16.444370 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:15:16.449392 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1317/2617] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:15:16 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:15:26.133364 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=50,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:15:40.861136 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cf6dd0-6772-48ab-b1c1-8ce968922593 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:15:40.862803 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-d3cf6dd0-6772-48ab-b1c1-8ce968922593 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:15:40.945535 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cf6dd0-6772-48ab-b1c1-8ce968922593 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:15:40.951885 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b36c142-0398-4a2b-a79c-73c5fd651c1a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e7a886715b5b4c1d99f0b37c58cb5417', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:15:40.952788 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b36c142-0398-4a2b-a79c-73c5fd651c1a service nova] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:15:40.953049 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b36c142-0398-4a2b-a79c-73c5fd651c1a service nova] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:15:40.953366 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b36c142-0398-4a2b-a79c-73c5fd651c1a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:15:40.955330 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-1b36c142-0398-4a2b-a79c-73c5fd651c1a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:15:40.969467 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b36c142-0398-4a2b-a79c-73c5fd651c1a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:15:40.971539 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b36c142-0398-4a2b-a79c-73c5fd651c1a service nova] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:15:40.985133 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1301/2618] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:15:40 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:15:49.069990 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461c98a6-69f8-4e62-89f5-19caa5a8c0a1 service placement] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:15:49.071454 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-461c98a6-69f8-4e62-89f5-19caa5a8c0a1 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:15:49.139695 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461c98a6-69f8-4e62-89f5-19caa5a8c0a1 service placement] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:15:49.144030 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575837ff-1ed5-4fcd-b9fe-7b00e65007f6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:15:49.144657 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575837ff-1ed5-4fcd-b9fe-7b00e65007f6 service neutron] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:15:49.144932 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575837ff-1ed5-4fcd-b9fe-7b00e65007f6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:15:49.145360 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575837ff-1ed5-4fcd-b9fe-7b00e65007f6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:15:49.147173 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-575837ff-1ed5-4fcd-b9fe-7b00e65007f6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:15:49.182895 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575837ff-1ed5-4fcd-b9fe-7b00e65007f6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:15:49.184382 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575837ff-1ed5-4fcd-b9fe-7b00e65007f6 service neutron] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:15:49.205357 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1318/2619] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:15:49 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:15:50.979441 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=34 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:15:53.408933 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b36c142-0398-4a2b-a79c-73c5fd651c1a service nova] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:15:53.419650 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b36c142-0398-4a2b-a79c-73c5fd651c1a service nova] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:15:53.424407 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bf55b8-e3c0-4390-80e6-c68d625a121b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81999329edfb4d4cac567d88d1f52909', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:15:53.425043 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bf55b8-e3c0-4390-80e6-c68d625a121b service neutron] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:15:53.425298 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bf55b8-e3c0-4390-80e6-c68d625a121b service neutron] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:15:53.425585 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bf55b8-e3c0-4390-80e6-c68d625a121b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:15:53.427633 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-c2bf55b8-e3c0-4390-80e6-c68d625a121b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:15:53.499138 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2bf55b8-e3c0-4390-80e6-c68d625a121b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:15:53.500671 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2bf55b8-e3c0-4390-80e6-c68d625a121b service neutron] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:15:53.511995 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1302/2620] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:15:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:15:56.672814 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575837ff-1ed5-4fcd-b9fe-7b00e65007f6 service neutron] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:15:56.674753 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-575837ff-1ed5-4fcd-b9fe-7b00e65007f6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:15:56.692117 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575837ff-1ed5-4fcd-b9fe-7b00e65007f6 service neutron] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:15:56.697986 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0670034b-ce28-424a-ae88-6a2ff523a8fa service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '63cc035057064805a95e9f12714ebade', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:15:56.698617 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0670034b-ce28-424a-ae88-6a2ff523a8fa service placement] REQUEST_METHOD: `GET` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:15:56.698893 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0670034b-ce28-424a-ae88-6a2ff523a8fa service placement] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:15:56.699164 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0670034b-ce28-424a-ae88-6a2ff523a8fa service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:15:56.704884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-0670034b-ce28-424a-ae88-6a2ff523a8fa service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:15:56.717882 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0670034b-ce28-424a-ae88-6a2ff523a8fa service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:15:56.720495 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0670034b-ce28-424a-ae88-6a2ff523a8fa service placement] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:15:56.733655 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1319/2621] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:15:56 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 30 01:16:00.010219 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bf55b8-e3c0-4390-80e6-c68d625a121b service neutron] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:16:00.011645 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-c2bf55b8-e3c0-4390-80e6-c68d625a121b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:16:00.065983 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bf55b8-e3c0-4390-80e6-c68d625a121b service neutron] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:16:00.070900 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a8b9f3-fe4b-4ba9-bcf3-dd9a71ea5f16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:16:00.071560 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a8b9f3-fe4b-4ba9-bcf3-dd9a71ea5f16 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:16:00.071981 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a8b9f3-fe4b-4ba9-bcf3-dd9a71ea5f16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:16:00.072252 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a8b9f3-fe4b-4ba9-bcf3-dd9a71ea5f16 admin admin] PATH_INFO: `/v3/users/96e0cd00e4ef4a2da960aa5d4f66972a` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:16:00.073498 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a8b9f3-fe4b-4ba9-bcf3-dd9a71ea5f16 admin admin] RBAC: Authorizing `identity:delete_user(user_id=96e0cd00e4ef4a2da960aa5d4f66972a)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:16:00.075441 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a8b9f3-fe4b-4ba9-bcf3-dd9a71ea5f16 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:16:00.130389 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b5a8b9f3-fe4b-4ba9-bcf3-dd9a71ea5f16 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '96e0cd00e4ef4a2da960aa5d4f66972a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:16:00.140346 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b5a8b9f3-fe4b-4ba9-bcf3-dd9a71ea5f16 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '96e0cd00e4ef4a2da960aa5d4f66972a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:16:00.140960 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b5a8b9f3-fe4b-4ba9-bcf3-dd9a71ea5f16 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '96e0cd00e4ef4a2da960aa5d4f66972a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:16:00.152314 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b5a8b9f3-fe4b-4ba9-bcf3-dd9a71ea5f16 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '96e0cd00e4ef4a2da960aa5d4f66972a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:16:00.165852 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-b5a8b9f3-fe4b-4ba9-bcf3-dd9a71ea5f16 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '96e0cd00e4ef4a2da960aa5d4f66972a'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:16:00.172427 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1303/2622] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:16:00 2022] DELETE /identity/v3/users/96e0cd00e4ef4a2da960aa5d4f66972a => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:16:00.354671 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0670034b-ce28-424a-ae88-6a2ff523a8fa service placement] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:16:00.368366 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0670034b-ce28-424a-ae88-6a2ff523a8fa service placement] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:16:00.426739 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6171735-9f88-4c68-b26a-a3cda6cfe424 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:16:00.427297 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6171735-9f88-4c68-b26a-a3cda6cfe424 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:16:00.427573 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6171735-9f88-4c68-b26a-a3cda6cfe424 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:16:00.427884 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6171735-9f88-4c68-b26a-a3cda6cfe424 admin admin] PATH_INFO: `/v3/projects/238d71be4e094cb6b84f9a48a8a2c79c` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:16:00.428985 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6171735-9f88-4c68-b26a-a3cda6cfe424 admin admin] RBAC: Authorizing `identity:delete_project(project_id=238d71be4e094cb6b84f9a48a8a2c79c)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:16:00.430896 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6171735-9f88-4c68-b26a-a3cda6cfe424 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:16:00.481034 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b6171735-9f88-4c68-b26a-a3cda6cfe424 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '238d71be4e094cb6b84f9a48a8a2c79c', 'request_id': 'req-b6171735-9f88-4c68-b26a-a3cda6cfe424'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:16:00.492029 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b6171735-9f88-4c68-b26a-a3cda6cfe424 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '238d71be4e094cb6b84f9a48a8a2c79c', 'request_id': 'req-b6171735-9f88-4c68-b26a-a3cda6cfe424'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:16:00.493278 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:16:00.500427 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b6171735-9f88-4c68-b26a-a3cda6cfe424 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '238d71be4e094cb6b84f9a48a8a2c79c', 'request_id': 'req-b6171735-9f88-4c68-b26a-a3cda6cfe424'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:16:00.512844 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b6171735-9f88-4c68-b26a-a3cda6cfe424 admin admin] The token cache is being invalidate because project 238d71be4e094cb6b84f9a48a8a2c79c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:16:00.513130 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b6171735-9f88-4c68-b26a-a3cda6cfe424 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:16:00.513955 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-b6171735-9f88-4c68-b26a-a3cda6cfe424 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:16:00.515582 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1320/2623] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:16:00 2022] DELETE /identity/v3/projects/238d71be4e094cb6b84f9a48a8a2c79c => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:16:00.519826 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a8b9f3-fe4b-4ba9-bcf3-dd9a71ea5f16 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:16:00.521420 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-b5a8b9f3-fe4b-4ba9-bcf3-dd9a71ea5f16 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:16:00.535994 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a8b9f3-fe4b-4ba9-bcf3-dd9a71ea5f16 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:16:00.570733 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:16:00.597311 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e27572-1252-4135-a3b5-c0f008e66c92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:16:00.597958 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e27572-1252-4135-a3b5-c0f008e66c92 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:16:00.598231 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e27572-1252-4135-a3b5-c0f008e66c92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:16:00.598485 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e27572-1252-4135-a3b5-c0f008e66c92 admin admin] PATH_INFO: `/v3/users/42de539be2df4b0082f01b3aa3746691` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:16:00.599670 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e27572-1252-4135-a3b5-c0f008e66c92 admin admin] RBAC: Authorizing `identity:delete_user(user_id=42de539be2df4b0082f01b3aa3746691)` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:16:00.601921 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e27572-1252-4135-a3b5-c0f008e66c92 admin admin] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:16:00.652903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-69e27572-1252-4135-a3b5-c0f008e66c92 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '42de539be2df4b0082f01b3aa3746691'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:16:00.661229 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-69e27572-1252-4135-a3b5-c0f008e66c92 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '42de539be2df4b0082f01b3aa3746691'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:16:00.661903 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-69e27572-1252-4135-a3b5-c0f008e66c92 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '42de539be2df4b0082f01b3aa3746691'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:16:00.669310 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-69e27572-1252-4135-a3b5-c0f008e66c92 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '42de539be2df4b0082f01b3aa3746691'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:16:00.681849 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.notifications [None req-69e27572-1252-4135-a3b5-c0f008e66c92 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '42de539be2df4b0082f01b3aa3746691'} {{(pid=81494) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:16:00.688223 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1304/2624] 10.222.0.16 () {58 vars in 1224 bytes} [Sun Oct 30 01:16:00 2022] DELETE /identity/v3/users/42de539be2df4b0082f01b3aa3746691 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:16:00.866630 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6171735-9f88-4c68-b26a-a3cda6cfe424 admin admin] Authenticating user token {{(pid=81495) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:16:00.868195 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.fernet_utils [None req-b6171735-9f88-4c68-b26a-a3cda6cfe424 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81495) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:16:00.881554 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6171735-9f88-4c68-b26a-a3cda6cfe424 admin admin] Validating token access rules against request {{(pid=81495) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:16:00.939356 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a513efde-55bb-4736-b7b2-ad97fb8493c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca6c0649e64b4d38bdd14a04f31beb73', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3907972f9f6248debeeda8ecd254820e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81495) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:16:00.940016 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a513efde-55bb-4736-b7b2-ad97fb8493c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:16:00.940289 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a513efde-55bb-4736-b7b2-ad97fb8493c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:16:00.940578 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a513efde-55bb-4736-b7b2-ad97fb8493c6 admin admin] PATH_INFO: `/v3/projects/39ed9d8c55cf4a549f2f49504650aa20` {{(pid=81495) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:16:00.941694 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a513efde-55bb-4736-b7b2-ad97fb8493c6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=39ed9d8c55cf4a549f2f49504650aa20)` {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:16:00.943802 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a513efde-55bb-4736-b7b2-ad97fb8493c6 admin admin] RBAC: Authorization granted {{(pid=81495) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:16:01.002625 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a513efde-55bb-4736-b7b2-ad97fb8493c6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '39ed9d8c55cf4a549f2f49504650aa20', 'request_id': 'req-a513efde-55bb-4736-b7b2-ad97fb8493c6'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:16:01.015132 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a513efde-55bb-4736-b7b2-ad97fb8493c6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '39ed9d8c55cf4a549f2f49504650aa20', 'request_id': 'req-a513efde-55bb-4736-b7b2-ad97fb8493c6'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:16:01.024410 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a513efde-55bb-4736-b7b2-ad97fb8493c6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '39ed9d8c55cf4a549f2f49504650aa20', 'request_id': 'req-a513efde-55bb-4736-b7b2-ad97fb8493c6'} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:16:01.037002 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a513efde-55bb-4736-b7b2-ad97fb8493c6 admin admin] The token cache is being invalidate because project 39ed9d8c55cf4a549f2f49504650aa20 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81495) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 30 01:16:01.037321 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a513efde-55bb-4736-b7b2-ad97fb8493c6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:16:01.037948 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG keystone.notifications [None req-a513efde-55bb-4736-b7b2-ad97fb8493c6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81495) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 30 01:16:01.039989 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: [pid: 81495|app: 0|req: 1321/2625] 10.222.0.16 () {58 vars in 1233 bytes} [Sun Oct 30 01:16:00 2022] DELETE /identity/v3/projects/39ed9d8c55cf4a549f2f49504650aa20 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 30 01:16:10.684870 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=36,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:16:11.033419 dv-cinder-master-3-iscsi devstack@keystone.service[81495]: DEBUG dbcounter [-] [81495] Writing DB stats keystone:DELETE=9,keystone:SELECT=56,keystone:INSERT=1 {{(pid=81495) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 30 01:20:22.261414 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e27572-1252-4135-a3b5-c0f008e66c92 admin admin] Authenticating user token {{(pid=81494) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 30 01:20:22.263553 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-69e27572-1252-4135-a3b5-c0f008e66c92 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:20:22.340490 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e27572-1252-4135-a3b5-c0f008e66c92 admin admin] Validating token access rules against request {{(pid=81494) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 30 01:20:22.405590 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8963394-dc6e-4ff9-8b07-dda6bf0a5010 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '63cc035057064805a95e9f12714ebade', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '658799ed31d2472fb36af4aff0f03aab', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81494) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 30 01:20:22.406203 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8963394-dc6e-4ff9-8b07-dda6bf0a5010 service placement] REQUEST_METHOD: `GET` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 30 01:20:22.406477 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8963394-dc6e-4ff9-8b07-dda6bf0a5010 service placement] SCRIPT_NAME: `/identity` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 30 01:20:22.406747 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8963394-dc6e-4ff9-8b07-dda6bf0a5010 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81494) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 30 01:20:22.408710 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.fernet_utils [None req-c8963394-dc6e-4ff9-8b07-dda6bf0a5010 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81494) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 30 01:20:22.420478 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8963394-dc6e-4ff9-8b07-dda6bf0a5010 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 30 01:20:22.422056 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8963394-dc6e-4ff9-8b07-dda6bf0a5010 service placement] RBAC: Authorization granted {{(pid=81494) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 30 01:20:22.432319 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: [pid: 81494|app: 0|req: 1305/2626] 10.222.0.16 () {60 vars in 1365 bytes} [Sun Oct 30 01:20:22 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 30 01:20:32.428116 dv-cinder-master-3-iscsi devstack@keystone.service[81494]: DEBUG dbcounter [-] [81494] Writing DB stats keystone:SELECT=58 {{(pid=81494) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}